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ABSTRACT 


Acquisition  program  risk  management  is  a  required  work  activity  throughout  an 
acquisition  programs  life  cycle.  A  program  manager  (PM),  in  today’s  acquisition  environment, 
must  continually  assess  program  risk  to  manage  program  uncertainty.  Risk  management  assists 
PMs  in  defining  if  they  can  meet  cost  (can  the  product  or  service  be  delivered  with  available 
funding  resources),  schedule  (can  the  product  or  service  be  delivered  in  time),  and  performance 
requirements  (whether  the  product  or  service  will  be  able  to  meet  mission-essential 
requirements).  Tools  for  effective  program  risk  management  are  widely  available  in  the  United 
States  commercial  sector.  The  Department  of  Defense  (DoD)  has  developed  and  published  a  Risk 
Management  Guide  for  DoD  Acquisition  (hereafter  referred  to  as  the  Guide).  The  Guide  provides 
a  tool  for  PMs  to  assess  risk  and  present  findings  to  senior-level  leaders  in  the  DoD  acquisition 
community. 

The  purpose  of  this  research  is  to  address  the  following  questions: 

•  Does  the  Guide  provide  an  effective  tool  in  managing  program  risk  in  today’s  acquisition 

environment? 

•  Can  the  Guide  be  improved? 

The  Strategy  Research  Project  consists  of  the  review  of  the  Guide  and  risk  management 
documentation  and  articles  from  multiple  Internet  sources.  The  Guide  is  presented  and  reviewed 
to  provide  the  reader  of  this  report  with  a  general  understanding  of  DoD  risk  management 
practices.  Risk  management  documentation  and  articles  provide  an  understanding  of  the 
effectiveness  and  usefulness  of  risk  management  practices. 

This  study  also  includes  a  survey  to  understand  risk  management  practices  currently  in 
use  by  the  DoD  acquisition  community.  The  survey  is  aligned  to  gather  data  on  knowledge  and 
relevance  of  the  Guide,  respondent  demographics,  and  other  risk  management  tools  now  used  by 
the  acquisition  community. 

DoD  acquisition  career  field  employees  are  the  target  population  for  this  study.  The  study 
is  cross-sectional  and  aimed  at  collecting  and  analyzing  data  one  time  for  this  population.  The 
results  and  conclusions  of  this  study  are:  The  study  identified  the  Guide  provides  a  basic  tool  for 
risk  management,  and  the  Guide  is  accepted  by  government  acquisition  personnel.  The  Guide 
does  not  provide  an  effective  tool  for  managing  the  wide  variety  of  projects  ongoing  in  the  DoD 
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acquisition  environment.  Many  recommended  improvements  to  the  Guide  were  identified 
through  this  research. 


CHAPTER  1 

INTRODUCTION  AND  RATIONAL 


Introduction 

Acquisition  program  risk  management  is  a  required  work  activity  throughout  an 
acquisition  program’s  life  cycle.  A  program  manager  (PM),  in  today’s  acquisition  environment, 
must  continually  assess  program  risk  to  manage  program  uncertainty.  Risk  management  assists  a 
program  (funding  resources),  schedule  (can  the  product  or  service  be  delivered  in  time),  and 
perfonnance  requirements  (whether  the  product  or  service  will  be  able  to  meet  mission  essential 
requirements).  Risk  management  is  becoming  more  and  more  critical  to  PMs  due  to  dwindling 
budgets  and  the  need  to  deliver  a  product  to  the  Services  within  acceptable  cost.  Programs  in 
today’s  acquisition  environment  can  be  canceled  readily  if  they  cannot  deliver  a  product  that 
perfonns  as  needed  by  the  user  within  available  program  dollars.  Most  Department  of  Defense 
(DoD)  acquisition  programs  continually  try  to  develop  products  that  push  the  edge  of 
technology,  require  state-of-the-art  computer  processing,  have  user  requirements  that  require 
operation  in  many  environmental  extremes,  and  have  demanding  availability  requirements.  So 
development  of  these  DoD  state-of-the-art  products  requires  a  full  understanding  of  program 
uncertainties  and  future  risks. 

PMs  are  required  to  balance  technical  perfonnance  against  available  funding.  In  a  PM’s 
world,  if  an  effort  requires  additional  time  or  development  activities,  additional  funding  usually 
will  be  required.  PMs  must  make  hard  decisions  if  additional  funding  is  not  available  to  complete 
the  program.  They  are  required  to  trade  product  perfonnance  against  available  funding,  and  that 
requires  complex  decisions  due  to  complex  products.  However,  product  perfonnance  in  all 
programs  has  a  baseline  and  in  DoD  the  baseline  is  defined  as  key  performance  parameters 
(KPPs).  All  programs  must  meet  the  KPPs,  or  the  product  may  not  be  usable  in  the  intended 
environment,  and  that  could  lead  to  cancellation  of  the  program. 

PMs  must  fully  understand  program  uncertainties  due  to  the  complexity  of  DoD 
acquisition  programs  and  today’s  acquisition  environment.  Doing  this  requires  a  full 
understanding  of  the  program  and  future  issues  that  may  develop.  This  can  be  a  difficult 
balancing  act  that  requires  full  analysis  of  the  product  and  a  projection  of  future  program 
uncertainties. 
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Tools  for  this  balancing  act  and  effective  program  risk  management  are  widely  available 
in  the  United  States  commercial  sector.  The  DoD  also  has  developed  and  published  a  Risk 
Management  Guide  for  DoD  Acquisition  (hereafter  referred  to  as  the  Guide)  programs.  The 
Guide  provides  a  tool  for  PMs  to  assess  program  risk  and  present  findings  to  senior-level  leaders 
in  the  DoD  acquisition  community. 

Background  of  the  Study 

The  Guide  provides  acquisition  personnel  a  basic  guide  for  the  assessment  and 
presentation  of  program  risks.  The  Guide  provides  a  process  for  risk  management,  key  activities 
for  risk  identification,  analysis  and  mitigation,  and  information  on  risk  planning  and  preparation 
activities  for  risk  management. 

The  Guide  is  the  DoD  effort  to  establish  a  baseline  for  the  management  and  reporting  of 
program  risk.  Overall,  the  Guide  presents  a  risk  management  matrix  that  projects  program  risk 
based  on  levels  of  likelihood  and  consequence  criteria. 

The  Guide  and  management  matrix  are  used  by  many  DoD  acquisition  programs  and 
presented  at  many  meetings  to  senior-level  leaders  as  the  tool  for  risk  management.  The  matrix 
along  with  rating  criteria  is  accepted  by  most  of  the  DoD  acquisition  community.  Many 
individuals  believe  the  Guide  is  rock-solid  for  risk  planning  and  presentation.  However,  the  risk 
management  activities  included  in  the  Guide  have  come  into  question  because  risk  assessment 
activities  are  considered  a  best-guess  activity  with  little  validity. 

Over  past  years,  during  multiple  meetings  and  program  reviews,  risk  reporting  in 
accordance  with  the  Guide  has  been  questioned  by  many  individuals  at  meetings  or  program 
reviews.  Questions  concerning  the  applicability,  reliability,  and  accuracy  of  the  data  presented 
based  on  the  Guide  can  sometimes  relate  to  the  process  for  risk  management  opposed  to 
program-related  risks.  Additionally,  risk  management  in  accordance  with  the  Guide  is  not  fully 
understood  by  all  DoD  organizations  and  individuals  outside  of  DoD  due  to  the  risk  matrix 
presentation.  If  individuals  or  organizations  do  not  fully  understand  the  basics  of  DoD  risk 
management,  an  understanding  of  the  infonnation  presented  in  the  risk  matrix  can  be  very 
confusing  for  an  individual  who  experiences  the  infonnation  for  the  first  time. 

Risk  management  is  critical  for  program  success.  Presentation  and  understanding  of 
program  risks  by  all  individuals  who  have  concerns  about  the  program  are  also  critical  for 
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program  success.  Having  the  right  tool  at  the  right  time  that  allows  full  definition  and 
understanding  of  program  risk  is  critical  for  successful  program  management. 

Problem  Statement 

The  Guide  provides  acquisition  personnel  a  basic  guide  for  the  assessment  and 
presentation  of  program  risks.  The  Guide  provides  a  process  for  risk  management,  key  activities 
for  risk  identification,  analysis  and  mitigation,  and  information  on  risk  planning  and  preparation 
activities  for  risk  management.  DoD  acquisition  programs  encounter  issues  with  cost,  schedule, 
and  perfonnance.  In  some  cases,  DoD  acquisition  programs  are  terminated  due  to  affordability  or 
the  lack  of  meeting  key  performance  requirements.  These  circumstances  indicate: 

1.  DoD  acquisition  programs  are  based  on  technology  that  is  not  ready  for  incorporation 
into  a  military  system. 

2.  User  requirements  for  material  solutions  are  beyond  what  can  be  developed  in  a  military 
system. 

3.  DoD  continually  develops  material  solutions  that  are  high-risk  endeavors. 

4.  Risk  management  tools  are  not  adequate  to  effectively  manage  risk  in  DoD  acquisition 
programs. 

5.  DoD  acquisition  program  schedules  and  budgets  are  initially  over-optimistic. 

This  research  will  investigate  circumstance  No.  4 — risk  management  tools  are  not 
adequate  to  effectively  manage  risk  in  DoD  acquisition  programs. 

Purpose  of  the  Study 

The  purpose  of  this  research  is  to  address  the  following  questions: 

•  Does  the  Guide  provide  an  effective  tool  in  managing  program  risk  in  today’s  acquisition 
environment? 

•  Can  the  Guide  be  improved? 

Significance  of  the  Study 

In  today’s  acquisition  environment,  risk  management  is  an  important  aspect  of  program 
management.  PMs  must  be  able  to  define  program  risk  and  assess  the  risk  in  accordance  with 
impacts  to  program  cost,  schedule,  and  perfonnance.  If  program  risk  is  not  assessed  and  tracked 
properly,  a  program  can  encounter  difficulties  that  may  lead  to  tennination.  Termination  of  a 
program  is  not  a  desired  outcome  for  DoD  acquisition.  All  products  in  the  acquisition  cycle  are 
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based  on  military  significant  need.  These  needs  are  intended  to  provide  new  equipment,  increase 
mission  capability,  and  above  all,  save  the  lives  of  our  service  members. 

The  preparation  for,  management  of,  and  presentation  of,  program  risk  is  one  of  the  most 
important  tasks  for  today’s  PM.  This  is  true  of  all  acquisition  programs  regardless  of  Acquisition 
Category  (ACAT)  Levels.  PMs  must  make  this  activity  a  must-do  to  ensure  they  are  able  to 
address  future  program  issues  and  have  a  plan  for  mitigation. 

This  study  is  intended  to  provide  insight  into  how  applicable  and  acceptable  the 
procedures  and  processes  presented  in  the  Guide  support  today’s  PMs  and  the  acquisition 
community. 

Research  Methodology  Overview 

This  research  follows  a  fonnal  systematic  application  of  a  scientific  method  to  study  an 
issue.  The  scientific  process  used  for  this  study  includes: 

•  Definition  of  the  issue 

•  Formulation  of  the  issue  hypotheses 

•  Collection  of  data 

•  Analysis  of  Data 

•  Statement  of  information  concerning  conclusions  and  confirmation  or  disconfirmation  of 

the  hypotheses. 

This  study  is  aligned  to  applied  research  and  the  solution  of  issues.  This  study  is 
conducted  for  the  purpose  of  evaluating  ongoing  instructions  for  DoD  acquisition  program  risk 
management.  This  study  follows  the  descriptive  research  method.  A  review  of  available 
applicable  data  and  a  survey  to  address  the  research  question  and  research  hypothesis  is  included. 
The  study  is  designed  to  measure  the  effectiveness  of  current  policy  in  relation  to  risk 
management. 

DoD  employees  in  the  acquisition  career  field  are  the  target  population  for  this  study.  The 
study  is  cross-sectional  and  aimed  at  collecting  and  analyzing  data  one  time  for  this  population. 

Research  Questions 

The  purpose  of  this  research  is  to  assess  if  the  Guide  provides  an  adequate  resource  for 
PMs  to  identify,  assess,  mitigate,  and  present  associated  program  risk.  Two  research  questions 
are  defined  to  develop  an  understanding  of  the  Guide  application  in  the  current  DoD  acquisition 
environment.  These  questions  are: 
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•  Does  the  Guide  provide  an  effective  tool  in  managing  program  risk  in  today’s  acquisition 
environment? 

•  Can  the  Guide  be  improved? 

Research  Hypothesis 

Two  hypotheses  for  this  research  are: 

•  HI:  The  Guide  process  and  procedures  does  not  provide  an  effective  tool  in  managing 
program  risk  in  today’s  acquisition  environment. 

•  H2:  Suggested  improvement  to  the  Guide  need  to  be  implemented. 

Research  Limitations 

Industry  and  DoD  risk  management  topics  are  widely  available  on  most  Internet  search 
engines.  Initial  search  results  indicate  there  are  more  than  12.9  million  articles  relating  to  DoD 
risk  management.  For  industry,  the  numbers  of  Internet  hits  are  staggering.  Initial  results  indicate 
there  are  more  than  42  million  articles  for  risk  management.  However,  most  of  the  articles  for 
both  the  DoD  and  Industry  are  oriented  to  the  discussion  and  application  of  risk  management. 
Very  few  are  related  to  the  assessment  of  how  well  the  risk  management  practices  help  or  hinder 
effective  program  management.  This  study  is  limited  to  articles  that  discuss  the  benefits  or 
detriments  of  the  Guide. 

Research  Questionnaire 

The  research  questionnaire  is  the  heart  and  soul  of  this  research  project.  The  survey  is 
designed  to  get  a  pulse  of  the  current  acquisition  community’s  feel  about  the  applicability  and 
effectiveness  of  the  Guide.  Considering  there  are  hundreds  of  thousands  of  individuals  working 
the  acquisition  of  programs,  there  is  a  limit  to  how  many  individual  surveys  can  be  analyzed.  The 
survey  is  designed  to  accommodate  input  from  many  acquisition  community  members  across 
functional  fields  but  represents  only  a  select  sample  of  all  involved  with  the  acquisition  of 
products  for  the  Joint  Services. 

Definition  of  Key  Terms  (DoD,  2006,  p.  33) 

Consequence:  The  outcome  of  a  future  occurrence  expressed  qualitatively  or 
quantitatively,  being  a  loss,  injury,  disadvantage,  or  gain  (DoD,  2006,  p.  33). 

Future  Root  Cause:  The  reason,  which,  if  eliminated  or  corrected,  would  prevent  a 
potential  consequence  from  occurring.  It  is  the  most  basic  reason  for  the  presence  of  a  risk  (DoD, 
2006,  p.  33). 
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Issue:  A  problem  or  consequence  which  has  occurred  due  to  the  realization  of  a  root 
cause.  A  current  issue  was  likely  a  risk  in  the  past  that  was  ignored  or  not  successfully  mitigated 
(DoD,  2006,  p.  33). 

Risk:  A  measure  of  future  uncertainties  in  achieving  program  performance  goals  within 
defined  cost  and  schedule  constraints.  It  has  three  components:  a  future  root  cause,  a  likelihood 
assessed  at  the  present  time  of  that  future  root  cause  occurring,  and  the  consequence  of  that 
future  occurrence  (DoD,  2006,  p.  33). 

Risk  Analysis:  The  activity  of  examining  each  identified  risk  to  refine  the  description  of 
the  risk,  isolate  the  cause,  and  detennine  the  effects,  and  aid  in  setting  risk  mitigation  priorities. 
It  refines  each  risk  in  terms  of  its  likelihood,  its  consequence,  and  relationship  to  other  risk  areas 
or  processes  (DoD,  2006,  p.  33). 

Risk  Identification:  The  activity  that  examines  each  element  of  the  program  to  identify 
associated  future  root  causes,  begin  their  documentation,  and  set  the  stage  for  their  successful 
management.  Risk  identification  begins  as  early  as  possible  in  successful  programs  and 
continues  throughout  the  life  of  the  program  (DoD,  2006,  p.  33). 

Risk  Management:  An  overarching  process  that  encompasses  identification,  analysis, 
mitigation  planning,  mitigation  plan  implementation,  and  tracking  future  root  causes  and  their 
consequence  (DoD,  2006,  p.  33). 

Risk  Management  Planning:  The  activity  of  developing  and  documenting  an  organized, 
comprehensive,  and  interactive  strategy  and  methods  for  identifying  and  tracking  future  root 
causes,  developing  risk-mitigation  plans,  performing  continuous  risk  assessments  to  determine 
how  risks  and  their  root  causes  have  changed,  and  assigning  adequate  resources  (DoD,  2006,  p. 
33). 

Risk  Mitigation  Plan  Implementation:  The  activity  of  executing  the  risk  mitigation  plan 
to  ensure  successful  risk  mitigation  occurs.  It  detennines  what  planning,  budget,  requirements, 
and  contractual  changes  are  needed,  provides  a  coordination  vehicle  with  management  and  other 
stakeholders,  directs  the  teams  to  execute  the  defined  and  approved  risk  mitigation  plans, 
outlines  the  risk  reporting  requirements  for  ongoing  monitoring,  and  documents  the  change 
history  (DoD,  2006,  p.  33). 

Risk  Mitigation  Planning:  The  activity  that  identifies,  evaluates,  and  selects  options  to  set 
risk  at  acceptable  levels  given  program  constraints  and  objectives.  It  includes  the  specifics  of 
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what  should  be  done,  when  it  should  be  accomplished,  who  is  responsible,  and  the  funding 
required  to  implement  the  risk  management  plan  (DoD,  2006,  p.  33). 

Risk  Tracking:  The  activity  of  systematically  tracking  and  evaluating  the  performance  of 
risk  mitigation  actions  against  established  metrics  throughout  the  acquisition  process  and 
develops  further  risk  mitigation  options  or  executes  risk  mitigation  plans,  as  appropriate.  It  feeds 
information  back  into  other  risk  management  activities  of  identification,  analysis,  mitigation 
planning,  and  mitigation  plan  implementation  (DoD,  2006,  p.  33). 
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CHAPTER  2 
LITERATURE  REVIEW 


Introduction 

The  purpose  of  this  study  was  to  determine: 

•  Does  the  Guide  provide  an  effective  tool  in  managing  program  risk  in  today’s  acquisition 
environment? 

•  Can  the  Guide  be  improved? 

A  literature  review  through  an  Internet  search  process  is  included  to  determine  if 
information  is  available  to  address  the  following  questions: 

•  Are  there  available  data  that  discusses  the  implementation  and  use  of  the  instructions 
provided  in  the  Guide? 

•  Are  there  any  benefits  for  the  use  and  implementation  of  the  instructions  provided  in  the 
Guide? 

•  Are  there  any  shortcomings  for  the  use  and  implementation  of  the  instructions  provided 
in  the  Guide? 

•  Are  there  any  suggest  improvements  to  the  Guide? 

Body  of  the  Literature  Review 

Managing  Risk  in  a  Program  Office  Environment  (Sheppard,  2003) 

The  author  of  this  article  provides  a  great  overview  of  the  importance  of  risk  management 
and  how  to  employ  a  risk  management  program.  The  article  follows  guidance  provided  in  the 
Guide.  The  article  provides  infonnation  on  why  risk  management  is  hard,  how  to  identify  risks, 
and  how  to  classify  and  present  program  risks. 

The  author  states:  An  effective  risk  management  program  can  provide  program  managers 
with  the  information  they  need  to  make  smart  decisions  in  the  face  of  this  uncertainty.  Although 
the  techniques  for  risk  management  are  well  documented  and  not  technically  difficult,  a  variety 
of  factors  make  them  hard  to  implement  effectively  (Sheppard,  2003,  p.  125). 

Three  things  make  effective  risk  management  hard: 

1.  It  seldom  seems  urgent.  It  deals — or  should  deal — with  events  far  enough  in  the  future 
that  there  is  sufficient  time  to  influence  the  situation  or  develop  alternatives. 
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Unfortunately,  less  important  daily  pressures  often  get  more  attention  (Sheppard,  2003,  p. 
125). 

2.  It  does  require  careful  thought.  People  have  to  understand  the  distinction  between  risks, 
which  have  a  degree  of  uncertainty  associated  with  them,  and  issues,  which  are  realities 
to  be  managed.  The  devil  is  in  the  details,  and  these  details  must  be  clearly 
communicated  to  isolate  the  uncertainty  and  understand  its  impact.  Understanding  the 
true  situation  will  allow  teams  to  focus  on  solving  the  right  problem  and  develop  far  more 
effective  mitigation  plans  (Sheppard,  2003,  p.  125). 

3.  It  requires  common  understanding  and  commitment  from  everyone  on  the  team.  This 
means  risk  management  must  be  part  of  the  organizational  culture,  with  strong  support 
from  senior  management  and  infonned  participation  by  the  entire  team.  Creating  that 
common  vision  and  institutionalizing  the  processes  takes  training,  an  investment  in 
resources,  and  occasional  reinforcement  (Sheppard,  2003,  p.  126). 

One  of  the  difficulties  with  the  risk  management  process  as  defined  by  the  author  is 
classification  of  risk  and  allocation  of  resources  to  mitigate  program  risks.  The  article  states  a 
nominal  classification  of  low,  medium,  or  high  may  be  useful  for  a  snapshot  of  risk  status  on  a 
program,  but  it  does  not  provide  enough  infonnation  to  allocate  resources.  Placing  risks  in  a  risk 
matrix  based  on  the  assessment  of  probability  and  impact  shows  their  relative  importance 
(ordinal  ranking),  but  this  still  does  not  quantify  the  dollarized  impact  to  the  program  or  justify  a 
level  of  risk  funding  to  fully  understand  the  risk  exposure  of  a  program,  the  cost  of  both  the 
impact,  and  the  mitigation  options  needs  to  be  assessed.  However,  these  assessments  cost  time 
and  money  and  should  be  reserved  for  those  risks  with  the  greatest  perceived  combination  of 
probability  and  impact  (Sheppard,  2003,  p.  132). 

The  article  provides  pointers  to  a  PM  for  an  effective  risk  management  program.  The 
author  states:  The  risk  manager  should  establish  the  process,  provide  training  for  the  team,  make 
it  easy  to  nominate  risks,  help  the  team  distinguish  between  risks  and  issues,  and  have  someone 
keep  records  of  progress  and  status  (Sheppard,  2003,  p.  135). 

Understanding  Risk  Management  in  the  DoD  (Bolles,  2003) 

The  intent  of  this  article  is  to  show  the  linkages  between  risk  management  and  contract 
administration.  The  author  does  identify  risk  management  is  mandatory  for  major  acquisition 
programs. 
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The  article  states: 

Although  the  Department  of  Defense’s  (DoD’s)  current  risk  management  direction 
presents  a  comprehensive  and  robust  approach  to  identifying,  assessing,  and  managing  risk,  it 
does  not  adequately  emphasize  the  interface  between  risk  management  and  contract 
administration. 

In  essence,  a  well-crafted,  risk-appropriate  contract  can  temper  the  sensitivity  between 
technical  risk  and  the  probability  of  cost  and  schedule  overruns,  while  a  poorly  crafted  contract 
can  actually  increase  the  probability  of  cost  and  schedule  overruns. 

By  better  linking  sound  risk  management  practices  with  sound  contract  administration 
practices,  the  DoD  stands  to  continue  being  the  bellwether  federal  agency  for  pushing  the  state- 
of-the-art  in  effective  risk  management.  There  is  no  dispute  that  there  is  a  strong  relationship 
between  technical  risk  and  cost  and  schedule  overruns,  nor  is  there  any  dispute  that  DoD  Project 
Offices  must  assess  and  mitigate  technical  risk  if  they  are  to  be  successful.  However,  what  must 
be  kept  in  mind  is  that  technical  risk  in  and  of  itself  does  not  directly  result  in  cost  and  schedule 
overruns. 

The  moderating  variable  is  the  manner  in  which  a  project’s  contract  is  crafted  and  how 
deftly  the  contract  is  administered,  given  the  nature  of  a  project’s  technical  risk.  In  essence,  a 
well-crafted,  risk-appropriate  contract  can  temper  the  sensitivity  between  technical  risk  and  the 
probability  of  cost  and  schedule  overruns,  while  a  poorly  crafted  contract  can  actually  increase 
the  probability  of  cost  and  schedule  overruns  (Bolles,  2003,  pg  141). 

The  article  defines  three  key  areas  where  DoD  guidance  is  lacking  infonnation  on  the 
relationship  between  risk  management  and  contract  management: 

1 .  The  DoD  guidance  offers  little  specificity  in  relating  the  nature  of  technical  risk  and  the 
appropriateness  of  one  contract  type  over  another.  For  example,  although  the  Defense 
Acquisition  University’s  (DAU)  Risk  Management  Guide  for  DoD  Acquisition  states  that 
“the  government  contracting  officer  should  select  the  proper  type  of  contract  based  on  an 
appropriate  risk  assessment,  to  ensure  a  clear  relationship  between  the  selected  contract 
type  and  program  risk”  (p.  32),  this  guidance  is  not  particularly  prescriptive  in  assisting  a 
Project  Office  in  choosing  the  most  appropriate  contract  type  vis-a-vis  the  results  of  a  risk 
assessment  (Bolles,  2003,  p.  143). 
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2.  The  DoD  guidance  does  not  discuss  the  relationship  between  contractor  and  government 
risk  sharing  arrangements  and  the  key  Federal  Acquisition  Regulation  (FAR)  clauses 
typically  invoked  in  a  contract  (Bolles,  2003,  p.  144). 

3.  The  DoD  guidance  only  addresses  only  risk  management  in  the  context  of  major  weapon 
systems  and  Automated  Information  Systems  (AIS)  acquisitions.  However,  the  Office  of 
Management  and  Budget  Circular,  which  is  the  governing  document  for  implementing 
risk  management  in  the  federal  government,  is  applicable  to  all  major  capital  asset 
acquisitions,  including  Military  Construction  (MILCON)  projects  and  environmental 
restoration  (ER)  projects.  As  such,  risk  management  should  be  as  much  a  component  of 
planning  for  and  managing  MILCON  and  ER  projects  as  it  is  for  weapon  systems  and 
AIS  projects  (Bolles,  2003,  p.  144). 

In  conclusion,  the  author  provides  four  critical  perspectives  for  improving  the  ties 
between  risk  and  contract  management.  Risk  management  is  an  extremely  powerful  component 
of  the  DoD’s  approach  to  procuring  major  capital  systems.  However,  the  current  DoD  direction 
could  be  improved  if  it  were  to  incorporate  a  more  robust  discussion  of  the  nexus  between  risk 
management  and  contract  administration.  Although  not  intended  to  be  the  final  word  on  this 
issue,  this  article  represents  an  attempt  to  raise  DoD  Project  Office  awareness  in  understanding 
this  critical  yet  misunderstood  issue.  To  recap,  DoD  Project  Offices  would  be  well  served  to: 

1)  Use  the  results  from  their  pre-acquisition  risk  analysis  to  choose  an  appropriate  contract 
vehicle  vis-a-vis  the  nature  of  the  risk  identified  in  the  analysis 

2)  Adopt  sound  risk  management  practices  for  all  major  acquisition  projects,  including 
MILCON  and  ER 

3)  Ensure  that  the  FAR  clauses  invoked  in  a  contract  are  congruent  with  the  risk  sharing 
arrangement  agreed  to  by  the  parties 

4)  And  ensure  that  the  Contracting  Officer  is  included  as  a  key  member  of  a  Project  Office’s 
risk  management  team  (Bolles  2003,  p.  151) 

Some  Considerations  for  Implementing  Risk  Management  in  Defense  Programs 
(Conrow  &  Fredrickson,  1996). 

However  dated,  this  article  still  provides  great  insight  into  the  DoD  risk  management 
practices  and  Guide.  The  article  focus  is  on  suggested  considerations  when  implementing  a  risk 
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management  program.  Many  deficiencies  are  identified  that  must  be  considered  in  risk 
management  practices. 

Key  points  of  the  article  include: 

•  The  risk  management  process  is  often  weakly  structured  or  “ad  hoc”  for  both  the 
government  and  contractors.  There  may  be  no  clearly  delineated  mechanism  in  place  for 
managing  program  risk  (e.g.,  organizational  responsibilities,  analyses,  products,  etc.),  or 
if  a  risk  management  process  exists,  it  may  be  present  on  paper  only.  The  risk  assessment 
portion  of  the  risk  management  process  is  often  too  subjective  and  not  adequately 
documented. 

•  The  prescribed  risk  assessment  categories  may  be  overly  broad  (e.g.,  management, 
technical),  leading  to  difficulty  in  evaluating  results  and  implementing  a  viable, 
measurable  risk  mitigation  strategy. 

•  A  weak  risk  assessment  methodology  may  be  used,  which  introduces  considerable  doubt 
as  to  the  accuracy  and  value  of  the  results  for  senior  management  use. 

•  Ordinal  risk  assessment  scales  are  often  incorrectly  applied.  Mathematical  operations 
cannot  be  applied  to  scores  obtained  from  uncalibrated  ordinal  risk  assessment  scales. 
Risk  values  generated  by  mathematical  operations  are  generally  meaningless  and  may 
hide  true  risk  issues. 

•  The  risk  assessment  results  may  be  summarized  into  broad  categories  (e.g.,  low,  medium, 
and  high)  without  sufficient  backup  to  understand  the  nature  of  the  risk  present. 

•  The  government  and  contractors  may  use  different,  incompatible  risk  assessment 
methodologies  making  comparing  results  difficult,  if  not  impossible.  The  emphasis  of  the 
risk  assessment  process  is  generally  on  the  uncertainty  associated  with  a  specific  event 
occurring,  with  less  attention  given  to  the  consequence  of  the  event  occurring.  Risk  is 
often  inaccurately  referenced  as  only  the  uncertainty  tenn.  However,  it  is  the  product  of 
the  uncertainty  and  consequence  terms  that  yields  risk.  In  addition,  both  the  uncertainty 
and  consequence  terms  require  evaluation  and  tracking  over  time.  Program  risk 
assessments  and  mitigation  plans  are  often  unlinked.  In  addition,  they  may  be  prepared 
on  an  as-needed  basis  with  limited  tracking  against  key  program  milestones  (Conrow  & 
Fredrickson,  1996,  pp.  6-11). 
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Embracing  Uncertainty  in  DoD  Acquisition  (Frick,  2010) 

The  author  of  this  article  presents  assumptions  and  improvements  about  risk  management 
beyond  other  author’s  recommendations.  The  author  presents  a  case  that  risk  management  should 
also  be  concerned  with  the  positive  side  of  risk  management  and  the  Guide  should  provide 
information  to  prepare  for  good  circumstances  that  can  benefit  the  program.  The  article  states: 
Uncertainty  is  an  inherent,  unavoidable  aspect  of  life  that  has  a  significant  impact  on  program  or 
project  management,  and  acquisition  in  general.  The  treatment  of  risk  management  within  the 
DoD  as  a  formal  element  of  acquisition  is  a  topic  discussed  extensively  in  the  acquisition 
profession.  DoD  fares  no  better  than  industry  in  the  number  of  projects  or  programs  that  fail  to 
meet  cost,  schedule,  or  perfonnance  baselines.  This  article  suggests  that,  overall,  the  DoD 
approach  to  uncertainty  is  flawed,  and  that  we  need  substantive  changes  to  the  structure  and 
policies  of  acquisition  to  become  more  effective  in  the  discipline  of  program  management. 

The  Guide  defines  risk  as  “a  measure  of  uncertainties  in  achieving  program  perfonnance 
goals  and  objectives  within  defined  cost,  schedule,  and  performance  constraints.”  However,  it  in 
no  way  implies  the  potential  positive  aspects  of  these  uncertainties.  The  tenns  used,  e.g., 
“schedule  slip,  budget  increase,  cannot  meet  key  program  milestones”  concentrate  only  on  the 
negative  aspects  of  uncertainty.  This  is  not  surprising.  The  Guide  specifically  states,  “While  such 
variation  could  include  positive  as  well  as  negative  effects,  this  guide  will  only  address  negative 
future  effects.  Most  of  us  tend  to  think  of  risk  solely  in  terms  of  negative  consequences.  Few 
academicians  or  organizations  even  address  the  positive  potential  of  uncertainty”  (Frick,  2010,  p. 
355). 

The  author  also  believes  a  risk  management  funding  reserve  should  be  pennitted  to 
support  the  PM  with  an  actionable  plan  to  reduce  risk.  The  author  states:  Both  control  and 
avoidance  assume  that  most  of  the  pitfalls  that  lead  to  potentially  increased  risks  have  been 
identified.  Plans  are  developed  to  identify  trigger  events  and  react  to  these  events  (control),  or 
actions  are  taken  to  reduce  the  number  of  items  (avoidance)  on  the  list.  In  contrast,  no  list  of  risk 
events  or  risk  triggers  will  be  comprehensive.  There  always  will  be  an  undefined  and 
unknowable  spectrum  of  unpleasant  things  that  can  happen.  Neither  of  these  approaches  (control, 
avoidance)  addresses  this  fact.  Assumption  covers  this  domain  of  the  unknowable — although  the 
Guide  does  not  acknowledge  this  purpose. 
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In  practice,  assumption  of  both  the  known  and  unknown  most  often  are  disingenuous 
pronouncements.  While  the  concept  of  a  management  reserve  is  a  well-established  practice  in 
industry,  I  have  yet  to  meet  a  single  government  PM  whose  reserve  survived  the  gauntlet  of 
program  reviews,  sweep  ups,  agency  taxes,  or  end-of-year  “unfunded  requirements.”  In  reality, 
management  reserves  seldom  exist  formally,  and,  if  they  do,  seldom  survive,  particularly  when 
fiscal  boundaries  are  relevant.  Unfortunate  events  result  in  schedule  slips,  cost  overruns,  or 
perfonnance  reductions.  In  practice,  baselines  are  adjusted  to  comport  with  reality,  or  the  number 
of  “required”  units  shrinks  to  meet  current  resources  (Frick,  2010,  p.  363). 

Book  review.  Effective  Risk  Management:  Some  Keys  to  Success,  2nd  Edition 

Author:  Edmund  H.  Conrow  (Wideman,  2003) 

The  author  of  this  book  review  completed  a  group  study  of  Effective  Risk  Management: 
Some  Keys  to  Success,  Second  Edition.  The  intent  of  the  study  was  to  present  information  on  an 
author  who  had  considerable  experience  with  the  development  of  the  DoD  Risk  Management 
Guide.  The  book  review  provides  a  number  of  quotable  statements  that  demonstrate  the  authors’ 
experience  and  orientation  to  program  risk  management: 

“One  of  the  key  implementation  issues  that  must  be  addressed  is  how  to  overcome  a 
corporate  culture  that  is  lacking  or  even  negative  toward  risk  management.” 

“I  would  also  be  remiss  if  I  did  not  say  that  risk  management  can  be  very  political  in 
some  programs.” 

“I  have  also  found  that  the  overall  effectiveness  of  a  risk  management  process  is 
primarily  detennined  by  two  factors,  namely,  technical  sophistication  and  implementation 
efficiency.” 

“Although  risk  avoidance  may  sometimes  be  the  best  risk  handling  approach,  program 
managers  should  not  expect  miracles  to  occur  on  demand  to  resolve  risk-related  issues  that 
should  have  been  properly  dealt  with  much  earlier”  (Wideman,  2003,  p.  3). 

Risk  Management  Considerations  for  Interoperable  Acquisition  (Meyers,  2006) 

The  author  of  this  technical  note  broadens  the  requirement  of  program  risk  management. 
The  focus  of  the  article  is  on  interoperable  acquisition,  which  includes  the  set  of  practices  that 
enable  acquisition,  development,  and  operational  organizations  to  collaborate  more  effectively  to 
field  interoperable  systems.  These  practices  are  achieved  through  sharing  relevant  information 
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and  perfonning  necessary  activities  that  enable  the  collective  behavior  of  these  organizations  to 
successfully  deliver  systems-of-systems  capabilities  (Meyers,  2006,  p.  1). 

The  author  discusses  systems  of  systems  acquisition  with  interoperability  requirements. 
The  author  compares  DoD  and  industry  risk  management  guides  to  interoperability  risk 
requirements.  Findings  indicate  information  provided  permits  risk  management  of  stovepipe 
acquisition  programs  and  does  not  address  overall  system  of  system  interoperability 
requirements.  Findings  of  the  report  state: 

•  The  current  specifications  related  to  risk  management  are  insufficient  to  achieve 
interoperable  risk  management.  For  example,  as  we  have  seen,  there  are  concepts  that 
are: 

-  insufficiently  specified  (e.g.,  the  relation  between  qualitative  and  quantitative  values 
of  probability) 

-  unstated  (e.g.,  the  identifier  of  a  risk  or  its  state)  (Meyers,  2006,  p.  23). 

•  Our  experience  indicates  current  methodologies  for  the  practice  of  risk  management  are 
insufficient  to  achieve  interoperable  risk  management.  Existing  practices  encapsulate 
behaviors  that  are  performed  with  regard  to  risk  management.  However,  the 
specifications  of  such  practices  do  not  address: 

-  Data  management  and  sharing  of  risk-related  data 

-  Behaviors  perfonned  in  a  collective  manner,  including  the  decision-making  process 
(Meyers,  2006,  p.  23). 

Conclusions  of  the  Literature  Review 

Industry  and  DoD  risk  management  topics  are  widely  available  on  most  Internet  search 
engines.  Initial  search  results  indicate  there  are  more  than  12.9  million  articles  relating  to  DoD 
risk  management.  For  industry,  the  numbers  of  Internet  hits  are  staggering.  Initial  results  indicate 
there  are  more  than  42  million  articles  for  risk  management.  However,  most  of  the  articles  for 
both  the  DoD  and  industry  are  oriented  to  the  discussion  and  application  of  risk  management. 
Very  few  are  related  to  the  assessment  of  how  well  the  risk  management  practices  help  or  hinder 
effective  program  management.  This  study  is  limited  to  a  few  articles  that  discuss  the  benefits  or 
detriments  of  the  Guide. 

The  intent  of  the  literature  review  was  to  determine: 
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•  Are  there  available  data  that  discusses  the  implementation  and  use  of  the  instructions 
provided  in  the  Guide? 

•  Are  there  any  benefits  for  using  and  implementing  the  instructions  in  the  Guide? 

•  Are  there  any  shortcomings  for  the  use  and  implementation  of  the  instructions  provided 
in  the  Guide? 

•  Are  there  any  suggested  improvements  to  the  Guide? 

These  answers  to  these  four  questions  are  as  follows: 

Are  there  available  data  that  discusses  the  implementation  and  use  of  the  instructions 
provided  in  the  Guide? 

Data  that  discusses  the  implementation  and  use  of  the  instructions  provided  in  the  Guide 
is  not  identified  in  the  literature  review.  Many  topics  that  discussed  the  Guide  are  included,  but 
specific  infonnation  in  relation  to  implementation  in  a  program  environment  is  not  available. 

Are  there  any  benefits  for  the  using  and  implementing  the  instructions  provided  in  the 

Guide? 

The  Guide  is  a  great  basic  source  of  instruction  for  instituting  a  risk  management 
program  in  today’s  acquisition  environment.  The  Guide  is  considered  one  of  the  DoD  and 
industry  standards  for  an  effective  risk  management  program.  The  Guide  does  provide  basic 
step-by-step  instructions  for  risk  management. 

Are  there  any  shortcomings  for  the  use  and  implementation  of  the  instructions  provided 
in  the  Guide? 

There  is  no  information  identified  that  described  shortcomings  of  the  information 
provided  in  the  Guide.  All  information  reviewed  indicated  acceptance  of  the  data  presented  in 
the  Guide.  However,  a  number  of  suggested  improvements  were  identified. 

Are  there  any  suggest  improvements  to  the  Guide? 

•  Nominal  risk  classification  of  low,  medium,  or  high  may  be  useful  for  a  snapshot  of  risk 
status  on  a  program,  but  it  does  not  provide  enough  infonnation  to  allocate  resources. 
(Sheppard,  2003,  p.  132).  The  current  Guide  needs  improvement  for  risk  classification. 
Infonnation  on  allocation  of  resources  and  program  consequences  based  on  risk  would 
benefit  PMs. 

•  The  cunent  Guide  could  be  improved  if  it  were  to  incorporate  a  more  robust  discussion 
of  the  nexus  between  risk  management  and  contract  administration  (Bolles  2003,  p.  141). 
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•  The  current  Guide  could  be  improved  if  a  discussion  of  the  relationship  between 
contractor  and  government  risk  sharing  was  included  (Bolles,  2003,  p.  144). 

•  The  current  Guide  could  be  improved  if  it  incorporated  a  discussion  on  the  tie  between 
government  and  contractor  risk  reporting  methods  (e.g.,  organizational  responsibilities, 
analyses,  products,  etc.)  (Conrow  &  Fredrickson,  1996,  pp.  6-1 1). 

•  The  current  Guide  could  be  improved  by  incorporating  a  discussion  on  the  positive  side 
of  risk  management.  The  Guide  should  provide  infonnation  to  prepare  for  good 
circumstances  that  can  benefit  the  program  (Frick,  2010,  p.  355). 

•  The  current  Guide  could  be  improved  if  it  were  to  incorporate  a  discussion  on 
interoperable  risk  management  (Meyers,  2006,  p.  23). 
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CHAPTER  3 
METHODS 


Introduction 

This  chapter  describes  the  research  perspective,  research  design,  and  research  questions 
and  hypotheses.  Information  concerning  participation,  population,  and  sample,  unit  of  analysis, 
research  instrument,  pilot  study,  data  collection  procedures,  data  collection  and  statistical 
analysis,  bias  and  error,  survey  validity  and  reliability  are  presented  in  this  chapter.  This  research 
follows  a  formal  systematic  application  of  a  scientific  method  to  study  an  issue.  This  study  is 
aligned  to  applied  research  and  the  solution  of  issues.  This  study  is  conducted  to  evaluate 
ongoing  instructions  for  DoD  acquisition  program  risk  management.  This  study  follows  the 
descriptive  research  method.  A  review  of  available  applicable  data  and  a  survey  are  included  to 
address  the  research  question  and  research  hypothesis.  The  study  is  designed  to  measure  the 
effectiveness  of  current  policy  in  relation  to  risk  management. 

Research  Perspective 

All  data  collected  in  this  study  are  designed  to  collect  both  qualitative  and  quantitative 
data  for  assessment  of  the  Guide  and  procedures.  The  main  purpose  of  this  study  is  to  gain 
knowledge  on  current  use,  applicability,  acceptability,  and  suggested  improvements  to  the  Guide. 
Members  throughout  the  acquisition  community  and  serving  in  different  functional  areas  are 
selected  to  ensure  input  is  acquired  from  as  many  perspectives  as  possible.  This  type  of 
information  and  study  input  is  intended  to  gain  a  slice  of  infonnation  from  the  broad  acquisition 
community. 

Research  Design 

This  research  includes  three  components.  The  first  component  includes  a  full  review  and 
presentation  of  the  Guide  to  ensure  readers  of  this  study  understood  the  process  and  procedures 
defined  by  the  Guide.  The  Guide  Review  is  included  as  Appendix  A.  The  second  component 
includes  a  review  of  applicable  literature  to  understand  if  other  sources  have  infonnation  on  the 
relevance  and  acceptability  of  the  instructions  included  in  the  Guide.  The  third  component  of  the 
study  is  designed  to  gain  infonnation  on  the  current  use,  applicability,  acceptability,  and 
suggested  improvements  to  the  Guide  from  acquisition  personnel  through  a  survey. 

This  three-component  design  is  pursued  to  gain  information  on  the  acceptance  of  the 
process  and  procedures  portrayed  by  the  Guide. 
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Research  Questions  and  Hypothesis 

Research  Questions: 

The  purpose  of  this  research  is  to  assess  if  the  Guide  provides  an  adequate  resource  for 
PMs  to  identify,  assess,  mitigate,  and  present  associated  program  risk.  Two  research  questions 
are  defined  to  develop  an  understanding  of  the  Guide  application  in  the  current  DoD  acquisition 
environment.  These  questions  are: 

•  Does  the  Guide  provide  an  effective  tool  in  managing  program  risk  in  today’s  acquisition 

environment? 

•  Can  the  Guide  be  improved? 

Research  Hypothesis: 

Two  hypotheses  for  this  research  are: 

•  HI:  The  Guide  process  and  procedures  do  not  provide  an  effective  tool  in  managing 

program  risk  in  today’s  acquisition  environment. 

•  H2:  Suggested  improvement  to  the  Guide  need  to  be  implemented. 

These  questions  are  aligned  to  address  the  intent  of  the  study:  Overall,  do  the  Guide  and 
instructions  in  the  Guide  provide  a  good  tool  for  risk  management,  or  are  there  any 
improvements  that  can  be  implemented  to  benefit  the  DoD? 

Participation,  Population,  and  Sample  Size 

Participation 

Participation  in  this  study  was  fully  voluntary  by  all  acquisition  personnel.  Acquisition 
personnel  were  selected  from  multiple  sources.  Individuals  were  not  selected  based  on  expected 
response  or  position.  The  infonnation  provided  was  not  designed  to  instruct  an  individual  on 
how  to  respond  to  the  survey  questions. 

Population 

Two  select  groups  were  chosen  for  the  survey.  The  first  group  included  individuals  from 
the  author’s  30-year  DoD  acquisition  experience.  All  these  individuals  were  not  selected  based 
on  experience  or  knowledge  of  the  DoD  risk  management  practices.  All  individuals  from  this 
group  were  selected  randomly  from  among  acquisition  personnel  ranging  from  the  very 
experienced  to  new  members  of  the  workforce  community.  The  second  group  selected  was  from 
current  and  past  participants  in  the  DAU’s  Senior  Service  College  Fellowship.  These  individuals, 
even  though  in  senior  acquisition  positions,  were  selected  based  on  the  wide  organizational  input 
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need  of  the  study.  The  study  was  to  have  a  dispersed  response  to  the  survey  by  individuals  in 
many  functional  areas.  Target  Functional  areas  were:  Program  Management,  Contracting, 
Information  Technology,  Life  Cycle  Logistics,  Production,  Quality  and  Manufacturing,  Systems 
Planning,  Research,  Planning  and  Engineering,  Test  and  Evaluation,  and  Requirements 
Management. 

Sample  Size 

•  First  Group:  106 

•  Second  Group:  16 
Unit  of  Analysis 

The  unit  of  analysis  for  this  study  is  a  small  sample  of  DoD  acquisition  personnel.  All 
individuals  and  organizations  could  not  be  sampled  due  to  the  sheer  size  of  the  acquisition 
community.  Select  members  from  multiple  organizations  are  included  from  two  populations.  The 
first  population  included  individuals  from  the  author’s  30  years  of  experience  in  DoD  acquisition 
and  the  second  population  included  personnel  from  DAU’s  Senior  Service  College  Fellowship 
program. 

Research  Instrument 

The  Guide  survey  includes  two  parts.  The  first  part  was  aligned  to  gain  demographic 
information  from  all  participants.  This  part  consists  of  seven  questions.  Demographic 
information  collected  includes  infonnation  on  the  participant’s  current  employer,  acquisition 
functional  area,  employment  type  and  pay  grade,  acquisition  certification  level,  years  employed 
in  current  position,  years  employed  in  the  DoD,  and  ACAT  Level  programs.  The  survey 
demographic  questions  were  designed  to  ensure  participants  could  not  be  identified  based  on 
their  response. 

The  second  part  of  the  survey  was  designed  to  ask  specific  questions  relevant  to  the 
Guide.  This  part  consisted  of  2 1  questions.  A  complete  review  of  the  Guide  was  included  in  this 
study  for  survey  question  development.  The  review  was  aimed  at  identifying  specific  questions 
to  allow  participant  feedback  on  the  Guide  and  suggested  improvements.  The  Guide  review  is 
included  as  Appendix  A.  Questions  1  through  5  of  the  survey  were  designed  to  gain  participant 
knowledge  on  the  Guide  and  use  of  the  Guide  by  the  Program  Office  or  Risk  Management  IPT. 
Questions  6  and  7  were  designed  to  understand  how  participants  define  and  track  risks  for  their 
program.  Questions  8  through  10  were  designed  to  gain  infonnation  on  the  acquisition  life  cycle 
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stage  of  the  participants’  programs  and  how  program  risks  are  updated  and  tracked.  Questions  1 1 
through  15  were  designed  to  gain  participant  assessment  of  the  Risk  Management  Reporting 
Matrix.  Questions  16  and  17  were  designed  to  arrive  at  an  understanding  if  root  cause  and  work 
breakdown  structure  risk  identification  were  used  by  the  participants  for  their  programs. 
Question  18  and  19  were  designed  to  determine  if  the  participants’  programs  have  a  risk 
management  and  risk  mitigation  plan  and  if  the  plan  is  in  accordance  with  the  Guide.  Question 
20  was  designed  to  gain  participants’  assessments  on  whether  the  Guide  is  a  useful  tool  for 
timely  and  accurate  decisions.  Questions  21  was  designed  to  understand  if  the  participants  use 
other  risk  management  tools  and  if  they  had  any  other  suggested  improvements  to  the  Guide. 

The  methods  for  survey  question  development  for  the  first  part  of  the  survey  were  based 
on  authors’  knowledge  of  demographic  infonnation  relevant  to  the  research  and  the  acquisition 
community.  The  methods  for  survey  question  development  for  the  second  part  of  the  survey 
were  based  on  the  review  of  the  Guide  for  risk  management  concepts  and  processes. 

Pilot  Study 

A  pilot  study  on  the  research  instrument  was  included  in  this  research.  The  survey  was 
reviewed  by  a  project  advisor  for  clarity,  content,  and  validity.  Additionally,  a  select  group  of 
individuals  who  are  very  familiar  with  risk  management  practices  reviewed  the  survey  for 
clarity,  content,  and  validity.  Their  recommendations  and  comments  were  included  in  the  survey. 
All  comments  were  administrative  and  clarifying  in  nature.  New  questions  or  removal  of 
questions  were  not  recommended. 

Data  Collection  Procedures 

The  survey  instrument  for  this  research  is  an  Internet  tool  titled  SurveyMonkey. 
SurveyMonkey  provides  a  great  tool  for  assembling  a  survey  from  multiple  participants.  The  risk 
management  survey  for  this  study  was  targeted  for  122  respondents.  SurveyMonkey  provided  the 
ability  to  collect  and  analyze  data  based  on  the  survey.  The  survey  was  sent  to  all  participants 
and  remained  open  for  3  weeks.  One  reminder  to  fill  out  the  survey  was  sent  at  the  start  of  the 
third  week  of  the  survey  to  ensure  all  individuals  had  a  chance  to  participate. 

Data  Collection  and  Statistical  Analysis 

The  data  were  collected  in  this  study  from  responses  from  all  individuals  surveyed. 
SurveyMonkey  provides  consolidation  of  the  results  from  the  survey.  The  program  includes  a 
data  analysis  section  that  can  present  data  in  table  or  figure  fonnats.  Data  for  this  research  are 
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more  easily  presented  in  table  format  due  to  design  of  the  questions  from  the  survey.  Many 
questions  included  the  option  to  provide  written  response  to  the  question.  These  responses  were 
collected  by  SurveyMonkey  and  presented  as  a  list  of  comments.  Many  of  the  relevant  comments 
are  presented  in  this  research.  Basic  statistical  analysis  of  the  responses  are  tabulated  and 
presented  to  provide  information  on  the  Guide.  Tables  in  this  research  from  SurveyMonkey 
include  three  basic  types  to  include  yes/no  table,  list  selection  table,  and  level  of  use  or 
knowledge  table.  Table  1  provides  an  example  of  the  yes/no  table.  These  tables  present 
information  on  answer  selected,  response  percentage,  and  response  count.  Table  2  provides  an 
example  of  the  list  selection  table.  These  tables  present  information  on  answer  selected,  response 
percentage,  and  response  count.  Table  3  provides  an  example  of  level  of  use  or  knowledge  table. 
This  type  of  table  allows  the  participant  to  select  a  level  of  knowledge  or  use  from  no  knowledge 
or  use  to  full  knowledge  or  use  in  a  scale  from  one  to  seven  selections.  The  table  provides 
information  on  the  participant’s  selection  on  the  one-to-seven  scale,  rating  average  and  response 
count. 


Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

No 

88.1% 

11.9% 

52 

7 

Table  1.  Yes/No  Table  Example 


Answer  Options 

Response 

Percent 

Response 

Count 

1-5  years 

52.5% 

42 

6-7  years 

23.8% 

19 

8-10  years 

2.5% 

2 

11-15  years 

7.5% 

6 

1 5+  years 

13.8% 

11 

Table  2.  List  Selection  Table 
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Answer 

Options 

Does 

not 

provide 

Provides 

Minimal 

Portrayal 

Fully 

Portrays 

Program 

Risk 

Rating 

Average 

Response 

Count 

DoD  Risk 

Management 

Guide 

2 

2 

1  10 

24 

17 

0 

4.84 

56 

Table  3.  Level  of  Use  or  Knowledge  Table 


Bias  and  Error 

Only  a  small  sampling  of  the  entire  DoD  acquisition  community  was  included  in  this 
research.  Many  organizations  and  personnel  who  may  have  a  different  interpretation  of  the  Guide 
are  not  included.  If  the  entire  DoD  acquisition  community  could  be  surveyed,  different  results 
might  be  found.  All  survey  participants  provided  input  voluntarily,  and  questions  were  designed 
for  ease  of  answer  based  on  knowledge  and  use  of  the  Guide.  Levels  of  knowledge  and  use  tables 
were  designed  subjectively.  Participant’s  responses  to  seven  different  selections  can  be  based  on 
personality,  program  acquisition  category,  or  good  or  bad  experience  with  the  Guide.  Results  of 
the  survey  are  not  handled  in  a  positive  or  negative  way.  All  results  are  reported  based  on 
participants’  responses.  Control  of  bias  and  error  in  future  research  may  be  controlled  through 
interviews  in  which  the  researcher  can  ask  additional  clarifying  questions. 

Survey  Validity  and  Reliability 

This  study  only  involves  a  small  group  of  the  acquisition  community.  All  members  of  the 
Joint  Service  acquisition  community  are  not  surveyed  due  to  the  sheer  number  of  expected  input. 
Acquisition  community  members  are  selected  from  an  appropriate  sample  of  individuals  in 
acquisition  positions  and  different  functional  areas.  This  approach  provides  a  great  first  step  in 
assessing  if  the  Guide  is  applicable  and  acceptable. 

Survey  validity  and  reliability  are  expected  to  be  high  due  to  the  basic  general  nature  of 
the  questions  developed  for  the  survey.  The  survey  research  and  the  responses  provided  are 
considered  valid  and  reliable  due  to  direct  correlation  to  the  information  provided  in  the  Guide. 
Summary 

In  summary,  the  methods  used  for  this  research  included  a  literature  review  and  a  survey 
to  address  the  research  questions  and  hypothesis.  The  study  included  a  review  of  the  Guide  for 
development  of  all  questions  in  the  survey.  The  overall  intent  of  the  research  was  to  assess  if  the 
Guide  and  practice  provides  a  good  tool  for  the  PM  in  today’s  acquisition  environment. 
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SurveyMonkey  was  used  as  the  survey  instrument  for  survey,  data  collection,  and  data  analysis. 
The  survey  was  sent  to  122  individuals  in  the  DoD  acquisition  community.  The  survey  included 
seven  demographic  questions  and  2 1  questions  aligned  to  collect  infonnation  on  the  Guide. 
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CHAPTER  4 
RESULTS 


Introduction 

Chapter  4  presents  and  discusses  the  results  of  DoD  Risk  Management  Survey.  The 
Chapter  is  divided  into  two  sections.  The  first  section  provides  information  collected  through  the 
survey  on  participant’s  demographic  information.  The  second  section  provides  participants 
responses  to  the  Guide  survey  questions  and  relative  comments. 

Survey  Results 

Section  1:  Participant’s  Demographic  Information: 

Question  1:  Current  employer: 

Eighty  participants  responded  they  were  employed  with  the  United  States  Army. 

Question  2:  What  is  your  Acquisition  Functional  Area? 

Seventy-nine  participants  responded  to  this  question.  Table  4  provides  a  response 
breakout  by  acquisition  functional  area. 


Answer  Options 

Response 

Percent 

Response 

Count 

Business,  Cost  Estimating,  and  Financial  Management 

1.3% 

1 

Contracting 

6.3% 

5 

Purchasing 

0.0% 

0 

Facilities  Engineering 

0.0% 

0 

Industrial  Property 

0.0% 

0 

Information  Technology 

0.0% 

0 

Life  Cycle  Logistics 

2.5% 

2 

Program  Management 

41.8% 

33 

Production,  Quality,  and  Manufacturing 

1.3% 

1 

Systems  Planning,  Research,  Development  and  Engineering 

31.6% 

25 

Test  and  Evaluation 

13.9% 

11 

Requirements  Management 

1.3% 

1 

International 

0.0% 

0 

Table  4.  Participants  Acquisition  Functional  Area 

Question  3:  What  is  your  employment  Type  and  Pay  Grade? 

Seventy-eight  participants  responded  to  this  question.  Table  5  provides  a  response 
breakout  by  employment  type  and  pay  grade. 
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Military 

Answer 

Options 

0-10 

0-9 

0-8 

0-7 

0-6 

0-5 

0-4 

Pay 

Grade 

0 

0 

0 

0 

0 

3 

1 

Civilian 

Answer 

Options 

SES 

GS-15/ 

equivalent 

GS-14/ 

equivalent 

GS-13  / 
equivalent 

GS-12/ 

equivalent 

GS-11/ 

equivalent 

GS-10/ 

equivalent 

Pay 

Grade 

1 

23 

20 

23 

4 

2 

1 

Table  5.  Participants  Employment  Type  and  Pay  Grade 

Question  4:  What  level  of  Acquisition  Certification  have  you  achieved  in  your 
career? 

Eighty  participants  responded  to  this  question.  Table  6  provides  a  response  breakout  by 
acquisition  certification  levels. 


Answer  Options 

Response 

Percent 

Response 

Count 

Level  1 

Level  II 

Level  III 

3.8% 

8.8% 

87.5% 

3 

7 

70 

Table  6.  Participants  Acquisition  Certification  Levels 

Question  5:  How  long  have  you  been  employed  in  your  current  position? 

Eighty  participants  responded  to  this  question.  Table  7  provides  a  response  breakout  by 
years  in  current  position. 


Answer  Options 

Response 

Percent 

Response 

Count 

1-5  years 

52.5% 

42 

6-7  years 

23.8% 

19 

8-10  years 

2.5% 

2 

11-15  years 

7.5% 

6 

1 5+  years 

13.8% 

11 

Table  7.  Participants  Years  in  Current  Position 
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Question  6:  How  long  have  you  been  employed  in  the  Department  of  Defense? 

Eighty  participants  responded  to  this  question.  Table  8  provides  a  response  breakout  by 
years  of  service  in  the  DoD. 


Answer  Options 


Response  Response 

Percent  Count 


I- 5  years 
6-7  years 
8-10  years 

II- 15  years 
1 5+  years 


10.0%  8 

7.5%  6 

5.0%  4 

10.0%  8 

67.5%  54 


Table  8.  Participants  Years  of  Service  in  the  Department  of  Defense 

Question  7:  Which  Acquisition  Category  programs  have  you  worked  (check  all  that 
apply)? 

Seventy-two  participants  responded  to  this  question.  Table  9  provides  a  response 
breakout  by  ACAT.  Responses  to  the  other  category  included  non  program  of  record  efforts.  As 
indicated  by  the  response  count,  many  participants  have  worked  on  multiple  ACAT  level 
programs. 


Answer  Options 

Response 

Percent 

Response 

Count 

ACAT  1 

33.3% 

24 

ACAT  II 

48.6% 

35 

ACAT  III 

73.6% 

53 

ACAT IV 

31.9% 

23 

Other  (please  specify) 

10 

Table  9.  Participants  ACAT  Level  Experience 

Section  2:  Participants  Responses  to  the  Guide  Survey  Questions  and  Relative 
Comments 

Question  1:  Are  you  familiar  with  the  Guide? 

Sixty-eight  participants  responded  to  this  question.  Table  10  provides  a  response 
breakout. 
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Answer 

Options 

DoD  Risk 

Not 

Familiar 

Some 

What 

Familiar 

Very 

Familiar 

Response 

Count 

Management 

Guide 

11 

3 

1  18 

10 

17 

8 

68 

Table  10.  Guide  Familiarity 


Question  2:  Do  you,  your  program  office,  or  you  in  a  supporting  role  to  the  program 
office  use  the  DoD  Risk  Management  process  provided  in  the  Guide? 

Fifty-nine  participants  responded  to  this  question.  Table  1 1  provides  a  response  breakout. 


Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

88.1% 

52 

No 

11.9% 

7 

Table  11.  Participants  Use  of  the  Guide 


Question  3:  Does  your  Program  Office  believe  Risk  Management  is  the 
responsibility  of  the  program  office  or  all  organizations  that  are  involved  in  your 
program? 

Fifty-six  participants  responded  to  this  question.  Table  12  provides  a  response  breakout. 


Answer  Options 

Response 

Percent 

Response 

Count 

Program  Office 

25.0% 

14 

All  Organizations 

75.0% 

42 

Table  12.  Risk  Management  Responsibility 


Question  4:  Does  your  Program  have  a  Risk  Management  IPT? 

Fifty-seven  participants  responded  to  this  question.  Table  13  provides  a  response 
breakout. 
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Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

36.8% 

21 

No 

63.2% 

36 

Table  13.  Risk  Management  IPT 


Question  5:  If  your  program  has  a  Risk  Management  IPT  who  is  on  the  IPT? 
(Check  all  that  apply.) 

Twenty-one  participants  responded  to  this  question.  Table  14  provides  a  response 
breakout.  Other  category  responses  did  not  relate  to  organization  or  individuals  who  support  the 
Risk  Management  IPT.  These  responses  provided  additional  information  on  other  answer 
options. 


Answer  Options 

Response 

Percent 

Response 

Count 

PM 

95.5% 

21 

Logistician 

59.1% 

13 

Tester 

63.6% 

14 

Evaluator 

50.0% 

11 

Requirements  Developer 

45.5% 

10 

User 

45.5% 

10 

Contracting  (Government) 

Contractor  (Product  Contractor  not  program  office  support 
contractor) 

36.4% 

63.6% 

8 

14 

Other  (please  specify) 

6 

Table  14.  Risk  Management  IPT  Membership 


Question  6:  The  Guide:  Does  your  Program  Office  or  Risk  Management  IPT  define 
program  risks  as?  (Please  check  all  that  apply.) 

Fifty-three  participants  responded  to  this  question.  Table  15  provides  a  response 
breakout.  Other  category  responses  related  to  cost,  schedule,  and  performance  risks  addressed  in 
the  next  question. 
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Answer  Options 

Response 

Percent 

Response 

Count 

Current  Program  Issues 

69.8% 

37 

Technology  Development  Requirement  Risks 

77.4% 

41 

Anticipated  Root  Cause  Future  Program  Risks 

69.8% 

37 

Other  (please  specify) 

5 

Table  15.  Risk  Management  Definition 


Question  7:  Does  your  Program  Office  or  Risk  Management  IPT  track  risks  for 
cost,  schedule,  or  performance?  (Please  check  all  that  apply.) 

Fifty- five  participants  responded  to  this  question.  Table  16  provides  a  response  breakout. 
Other  category  responses  did  not  relate  to  the  question. 


Answer  Options 

Response 

Percent 

Response 

Count 

Cost 

Schedule 

Performance 

96.4% 

98.2% 

100.0% 

53 

54 

55 

Table  16.  Risk  Management  Tracking 


Question  8:  What  is  the  acquisition  life  cycle  stage  of  your  program  (please  check 
one)? 

Fifty-eight  participants  responded  to  this  question.  Table  17  provides  a  response 
breakout.  Other  category  responses  indicated  the  participants  are  working  in  an  acquisition 
environment  that  has  programs  in  multiple  stages  of  the  life  cycle  or  are  working  on  non  program 
of  record  efforts. 


Answer  Options 

Response 

Percent 

Response 

Count 

Material  Solution  Analysis 

6.5% 

3 

Technology  Development 

17.4% 

8 

Engineering  and  Manufacturing  Development 

32.6% 

15 

Production  and  Deployment 

32.6% 

15 

Operations  and  Support 

10.9% 

5 

Other  (please  specify) 

12 

Table  17.  Program  Acquisition  Life  Cycle  Stage 
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Question  9:  When  does  your  Program  Management  Team  or  Risk  Management  IPT 
update  program  risks  (check  all  that  apply)? 

Fifty-seven  participants  responded  to  this  question.  Table  18  provides  a  response 
breakout.  Other  category  responses  indicated  program  risks  are  updated  continuously. 


Answer  Options 

Response 

Percent 

Response 

Count 

Monthly 

44.4% 

24 

Quarterly 

35.2% 

19 

Semi-annually 

14.8% 

8 

Milestones 

24.1% 

13 

Critical  Program  Events 

44.4% 

24 

Other  (please  specify) 

3 

Table  18.  Program  Risks  Update 


Question  10:  What  risk  management  tracking  tool  does  your  Program  Management 
Team  or  Risk  Management  IPT  use?  (Please  check  all  that  apply.): 

Sixty  participants  responded  to  this  question.  Table  19  provides  a  response  breakout. 
Other  category  responses  provided  additional  information  on  how  the  tools  are  used  to  present 
program  risks. 


Answer  Options 

Response 

Percent 

Response 

Count 

Excel  spreadsheet 

Word  document 

PowerPoint  Presentation 

Other  (please  specify) 

45.1% 

47.1% 

78.4% 

23 

24 

40 

9 

Table  19.  Risk  Management  Tracking  Tools 


Question  11:  Do  you  believe  the  Risk  Management  Reporting  Matrix  provides  an 
accurate  and  fair  portrayal  of  program  risk  based  on  likelihood  and  consequence? 

Fifty-six  participants  responded  to  this  question.  Table  20  provides  a  response  breakout. 
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Answer 

Options 

Does 

not 

provide 

Provides 

Minimal 

Portrayal 

Fully 

Portrays 

Program 

Risk 

Rating 

Average 

Response 

Count 

DoD  Risk 

Management 

Guide 

2 

2 

1  10 

24 

17  0 

4.84 

56 

Table  20.  Risk  Management  Matrix  Accuracy  and  Portrayal 


Question  12:  Do  you  agree  the  Guide  level  of  likelihood  instruction  provides  a  fair 
assessment  of  risk  likelihood  and  probability  of  occurrence?  If  not,  please  explain 
why  and  what  improvements  should  be  considered. 

Fifty-five  participants  responded  to  this  question.  Table  21  provides  a  response  breakout. 
Five  improvement  comments  were  received  that  directly  relate  to  suggested  improvements  as 
follows: 

•  Current  tools  need  additional  space  for  descriptive  texts  on  likelihood  and  consequences. 
Some  risks  have  multiple  potential  outcomes  and  need  to  be  described  as  such. 

•  The  process  (colored  chart)  is  too  simplified  and  to  open  for  interpretation.  There  needs 
to  be  a  more  detailed  process  for  all  program  areas  (financial,  technical,  operational,  etc.). 

•  All  risks  charts  should  include  a  narrative  specifying  the  rational  for  the  assessment. 

•  A  proper  assessment  of  Program  Probability  of  Success  has  to  have  a  Monte  Carlo 
analysis  and  a  Risk  Managers  assessment  of  the  integrated  (c,s,p)  risk  for  the 
probabilities  to  mean  anything.  Probabilities  of  individual  risks  do  not  provide  the 
comprehensive  view,  and  give  a  false  picture  of  what  the  real  risk  driver  in  the  program 
is.  Couple  the  individual  risks  to  an  objective  analysis  such  as  Monte  Carlo. 

•  The  Guide  needs  to  be  more  clearly  defined  in  the  evaluation  of  the  risks. 


Answer 

Options 

DoD  Risk 

Does 

not 

provide 

Provides 

Minimal 

Portrayal 

Fully 

Provides 

Likelihood 

Rating 

Average 

Response 

Count 

Management 

Guide 

2  1 

0 

11 

23 

18 

0 

4.93 

55 

Suggested  Improvements 


Table  21.  The  Guide  Level  of  Likelihood  Instruction 
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Question  13:  Has  your  Program  Office  or  Risk  Management  IPT  ever  modified  or 
customized  the  level  of  likelihood  for  a  specific  program?  If  so,  please  explain: 

Sixty-three  participants  responded  to  this  question.  Table  22  provides  a  response 
breakout.  A  review  of  the  modification  and  customization  comments  did  not  provide  any 
substantial  or  reportable  information. 


Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

15.9% 

10 

No 

84.1% 

53 

Modification  or  Customization 

10 

Table  22.  Level  of  Likelihood  Modification 


Question  14:  Do  you  agree  the  Guide  Levels  and  Type  of  Consequence  Criteria 
provide  a  fair  assessment  of  risk  consequence?  If  not,  please  explain  why  and 
improvements  that  should  be  considered. 

Sixty-one  participants  responded  to  this  question.  Table  23  provides  a  response  breakout. 
Seven  improvement  comments  were  received  that  directly  related  to  suggested  improvements  as 
follows: 

•  Current  tools  need  additional  space  for  descriptive  texts  on  likelihood  and  consequences. 
Some  risks  have  multiple  potential  outcomes  and  need  to  be  described  as  such. 

•  My  issue  with  the  Guide  and  Risk  Management  in  general  is  this,  you  focus  on  what  can 
and  will  go  wrong  instead  of  what  can  and  will  go  right.  Research  has  shown  that 
focusing  on  the  negative  tends  to  lead  to  negative  results.  It  is  very  deficit  on  the  risk- 
reward  side  of  things  and  gets  people  to  look  at  how  to  fail  rather  than  how  to  succeed.  I 
believe  there  is  always  a  way  to  succeed.  You  just  have  to  find  it. 

•  It  still  leaves  to  much  room  for  individual  interpretation. 

•  The  consequences  can  vary  significantly  within  a  risk  area. 

•  Criterions  are  too  broad  and  do  not  look  at  a  comprehensive  view  of  the  program.  The 
criteria  do  not  define  pre-mitigated  or  post-mitigated  stance.  Example:  Just  because  an 
individual  risk  consequence  says  the  program  breaches  APB  by  6  months,  it  does  not 
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necessarily  mean  that  will  happen.  It  depends  on  the  PM’s  probability  and  comprehensive 
risk  analysis  to  the  Program,  not  individual  risks. 

•  The  previous  editions  of  the  Guide  provided  casualty  and  dollar  values  for  applying 
consequence  assessment.  This  was  not  an  effective  way  of  trying  to  quantify  consequence 
risk.  The  newer  version  gives  a  scaling  percentage  depending  on  the  dollar  amount  of  the 
program — a  huge  improvement.  Still,  no  two  programs  are  identical  and  should  tailor 
their  thresholds  for  accepting  consequences,  depending  on  the  overall  risks  associated 
with  the  program.  Also,  it  can  be  difficult  to  balance  what  the  final  value  should  be  after 
balancing  cost,  schedule,  and  performance. 

•  The  consequence  determination  does  not  reflect  how  various  portions  of  the  acquisition 
cycle  operate.  It  does  not  allow  proper  flexibility  to  trade  cost  vs.  Schedule,  nor  does  it 
allow  for  80  percent  solutions  that  will  be  accepted  in  the  end. 


Answer 

Options 

Does 

not 

provide 

Provides 

Minimal 

Assessment 

Fully 

Provides 

Assessment 

Rating 

Average 

Response 

Count 

DoD  Risk 

Management 

Guide 

0  2 

3 

14  26 

16  0 

4.84 

61 

Concerns  and  Suggested  Improvements 

15 

Table  23.  The  Guide  Levels  and  Type  of  Consequence  Criteria 


Question  15:  Has  your  Program  Office  or  Risk  Management  IPT  ever  modified  or 
customized  the  consequence  criteria  for  a  specific  program?  If  so,  please  explain. 

Sixty-two  participants  responded  to  this  question.  Table  24  provides  a  response  breakout. 


Consequence  Criteria  Modification  provided  by  the  participants  was  not  applicable  to  the 


question. 


Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

No 

Consequence  Criteria  Modifications 

16.1% 

83.9% 

10 

52 

10 

Table  24.  The  Guide  Consequence  Criteria 
Modification  or  Customization 
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Question  16:  Do  you  believe  the  Guide  provides  sufficient  information  to 
understand  and  assess  Root  Cause  Program  Risks?  If  not,  please  explain  why  and 
improvements  that  should  be  considered. 

Sixty-two  participants  responded  to  this  question.  Table  25  provides  a  response  breakout. 
Seven  improvement  comments  were  received  that  directly  relate  to  suggested  improvements  as 
follows: 

•  No,  I  believe  the  Guide  is  focused  on  detennining  the  severity  of  the  problem.  I  do  not 
believe  it  is  a  tool  for  detennining  the  root  cause.  Our  program  was  in  the  EMD  phase 
and  had  technology  readiness  and  engineering  issues.  I  am  not  sure  how  the  Guide  could 
be  modified  to  understand  the  root  causes  of  these  types  of  issues. 

•  The  risks  tools  provided  do  not  lend  themselves  to  identifying  root  causes.  They 
generally  describe  the  potential  risks  in  terms  of  potential  symptoms  and  remedies.  Some 
program  risks  for  financial  and  schedule  items  are  political  in  nature  and  externally 
driven.  Some  are  process  driven  based  on  interpretation  of  statutes  and  regs.  Root  cause 
analysis  wouldn’t  help  in  these  situations  but  would  identify  the  real  issues  to  decision 
makers  and  permit  the  full  story. 

•  It  would  be  nice  if  the  Guide  included  techniques  to  perfonn  root  cause  analysis,  such  as 
fishbone  diagrams,  etc. 

•  Does  not  adequately  portray  advocacy  of  your  program  among  stakeholders  and  leaders. 

•  Does  not  adequately  require  an  analysis  of  changes  in  the  strategic  environment  that  may 
negatively  impact  the  program. 

•  Root  cause  analysis  really  isn’t  a  main  focus  of  the  Guide,  although  it  is  important 
because  more  cursory  risks  could  not  be  the  true  source  of  the  problem. 

•  Without  identifying  and  managing  the  root  cause,  the  cursory  risks  may  not  be 
manageable.  A  possible  addition  to  the  Guide  could  be  using  the  Ishikawa  fishbone 
diagram  in  conjunction  with  the  standard  consequence  and  likelihood  assessment.  It’s  a 
simple  method  and  identifies  the  risk  for  likelihood  and  consequence  assessment. 

•  I  think  the  Guide  and  the  tools  available  are  adequate,  but  most  people  need  more  training 
in  how  to  actually  define  a  risk  and  how  to  figure  out  the  root  cause. 


37 


Answer 

Options 

Does 

not 

provide 

Provides 

Minimal 

Information 

Provides 

Required 

Information 

Rating 

Average 

Response 

Count 

DoD  Risk 

Management  2  2 

Guide 

Root  Cause  Improvements 

3 

19 

18 

17  1 

4.68 

62 

16 

Table  25.  Root  Cause  Program  Risks 


Question  17:  Do  you  agree  the  Work  Breakdown  Structure  (WBS)  process  for  risk 
identification  as  presented  in  the  Guide  is  sufficient  for  Root  Cause  Analysis  and 
Risk  Identification?  If  not,  please  explain  why  and  improvements  that  should  be 
considered. 

Fifty-nine  participants  responded  to  this  question.  Table  26  provides  a  response  breakout. 

Six  improvement  comments  received  directly  relate  to  suggested  improvements  as  follows: 

•  WBS  process  is  one  of  many  tools,  but  it  alone  won’t  identify  all  risks.  It  doesn’t  identify 
all  root  causes  because  they  aren’t  always  related  to  a  WBS. 

•  A  good  start  but  not  all  encompassing. 

•  The  Guide  mentions  going  to  a  WBS  level  4  or  5  element,  which  in  turn,  could  have 
several  root  causes.  There  should  be  some  distinction  in  the  Guide  between  different 
ACAT  levels.  For  example,  for  programs  which  I  am  familiar  with,  ACAT  II-IV,  EVMS 
[Earned  Value  Management  System]  is  only  captured  at  Level  2  of  the  WBS.  Also,  the 
WBS  may  lead  the  IPT  or  program  office  to  some  other  factor.  The  IPT  or  program  office 
may  be  better  identifying  risks  through  the  KPPs,  the  threshold  vs.  objective  requirements 
in  the  Performance  Specification,  or  some  other  methodology. 

•  This  is  the  best  tool  if  you  could  go  down  the  WBS  to  a  very  deep  level,  but  this  could  be 
labor  intensive  and  not  possible  to  complete  for  all  risks.  Overall,  the  process  could  cost 
more  than  the  project  and  provide  no  cost  savings — one  of  the  goals  of  risk  management. 
I  would  consider  using  a  more  global  approach  to  identify  root  causes  and  identify  risks, 
including  an  overarching  IPT  for  the  project  as  a  whole. 

•  This  depends  on  the  level  of  the  WBS  and  how  well  developed  the  WBS  is.  There  are 
interdependencies  that  can  be  missed  by  just  looking  at  the  WBS. 
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Your  risk  identification  is  only  as  good  as  your  WBS.  Although  this  technique  can  be 
applied  directly  to  a  contract  and  identify  contract  risk,  there  are  other  areas  of  a  program 
that  contain  risk  that  does  not  follow  a  “WBS”  structure  to  level  needed  to  identify  risk. 


Answer  Options 

Do  not 
agree 

Some 

What 

agree 

Fully 

Agree 

Rating 

Average 

Response 

Count 

DoD  Risk 

Management  2 

Guide 

Suggested  Improvements 

0 

1 

24 

21 

10 

1 

4.63 

59 

14 

Table  26.  Work  Breakdown  Structure  for  Root  Cause  Analysis 

Question  18:  Does  your  Program  Office  or  Risk  Management  IPT  have  a  Risk 
Mitigation  Plan? 

If  so,  does  the  plan  follow  the  suggested  Guide  format? 

Fifty-five  participants  responded  to  this  question.  Table  27  provides  a  response  breakout. 


Does 

Uses 

Answer  Options 

Not 

Follow 

Parts  of 
the 

Follows 

Format. 

Rating 

Average 

Response 

Count 

Format 

Format 

DoD  Risk 

Management  Guide 

5 

1 

3 

17 

14 

11 

4 

4.51 

55 

answered  question 

55 

Table  27:  Risk  Mitigation  Plan 


Question  19:  Does  your  Program  Office  or  Risk  Management  IPT  have  a  Risk 
Management  Plan?  If  so,  does  the  plan  follow  the  suggested  Guide  format? 

Fifty-eight  participants  responded  to  this  question.  Table  28  provides  a  response 
breakout. 


Answer  Options 

Does  not 

Follow 

Format 

Uses 

Parts  of 
the 

Format 

Follows 

format! 

Rating 

Average 

Response 

Count 

DoD  Risk 

Management  Guide 

4  2 

2 

20  13 

12  5 

4.59 

58 

Table  28.  Risk  Management  Plan 
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Question  20:  Do  you  believe  the  Guide  Risk  Reporting  Process  provides 
management  with  the  necessary  information  to  make  timely  and  accurate  decisions? 

Sixty-one  participants  responded  to  this  question.  Table  29  provides  a  response  breakout. 


Answer 

Options 

DoD  Risk 

Does 

not 

Provide 

Provides 

Some 

Information 

Provides  all 
Information 

Rating 

Average 

Response 

Count 

Management 

Guide 

3  2 

3 

22 

20 

10  1 

4.44 

61 

Table  29.  Risk  Management  Information 


Question  21:  Does  your  organization  use  any  other  Risk  Management  tool  to  assess 
program  risk?  If  so,  please  provide  a  description  and  if  possible  the  tool. 

Sixty-six  participants  responded  to  this  question.  Table  30  provides  a  response  breakout. 
One  improvement  comment  was  received  that  directly  relates  to  suggested  improvements  as 
follows: 

•  Currently  evaluating/using  a  systems  engineering  process  to  help  with  identifying 
technical  risks.  It  includes  spider  charts,  gaps  analysis,  etc. 


Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

No 

Other  Tool  Description 

19.7% 

80.3% 

13 

53 

14 

Table  30.  Other  Risk  Management  Tools 


Question  22:  Do  you  have  any  recommended  improvements  to  the  Guide? 

Thirty-three  participants  responded  to  this  question.  Nine  improvement  comments  were 
received  that  directly  relate  to  suggested  improvements  as  follows: 

•  Our  risk  management  plan  was  fairly  good  at  identifying  risk  and  developing  plans  to  fix 
the  issue.  The  problem  is  the  get-well  plans  were  not  funded.  Funding  needs  to  be  made 
available  to  make  the  process  effective. 

•  Change  the  focus  to  success  rather  than  failure. 
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•  I  think  the  Guide  is  decent,  but  the  paradigm  is  to  not  report  “red”  risk  items.  High-risk 
items  get  downgraded  to  medium  even  if  they  are  still  high.  DoD  management  needs  to 
come  to  grips  with  the  reality  of  high-risk  items  and  ways  to  track  and  mitigate  them.  No 
one  wants  to  present  a  high-risk  item  due  to  the  perception  that  the  program  would  be  in 
danger  of  losing  funding. 

•  I  would  suggest  decoupling  the  technical  perfonnance,  schedule,  and  cost,  in  the 
consequence  matrix  to  arrive  to  a  combined  score  of  the  three  separate  elements,  instead 
of  using  a  simplified  single  score  where  all  three  elements  are  tied  together.  This  will 
allow  for  a  more  precise  evaluation  of  the  consequences. 

•  No  Guide  is  adequate  if  the  MDA  and  PM  doesn’t  require  its  use.  Tighten  up  the  DoDI 
5000.02  and  compliance  will  be  better. 

•  My  recommendations  would  be  to  reassess  and  clearly  define  the  risk  and  consequences 
in  the  Guide. 

•  Most  programs’  risks  are  subjectively  detennined  and  documented  by  the  materiel 
developer.  Suggest  combat  developer  and  evaluator  input  to  risk  of  threat  changing  and 
the  potential  of  threshold  requirements  not  being  met. 

•  Provide  more  guidance  details  on  tracking  process.  It  seems  our  PM/programs  do  a  good 
job  identifying  risk  areas  and  preparing  risk  management  plans  but  not  in  “measuring”  or 
tracking  or  mitigating  the  risk.  Risk  management  on  programs  becomes  more  of  an 
exercise  to  go  through  (check  the  box)  rather  than  an  effective  tool  for  mitigating, 
planning  for,  managing,  monitoring,  and  reducing  risk. 

•  Need  more  training  on  risk  management  and  need  to  insure  that  government  agencies  are 
using. 

Summary  of  Results 

Participation  in  the  Guide  Survey  was  great.  The  survey  was  sent  to  122  individuals,  and 
81  (66.3  percent)  responses  were  received.  Responses  to  each  question  in  the  survey  averaged  55 
(45  percent)  to  more  than  60  (49  percent).  This  type  of  response  to  an  Internet-based  random 
survey  is  exceptional.  This  result  may  be  attributable  to  the  subject  matter  and  interest  of  the 
DoD  acquisition  community. 

All  participants  identified  themselves  as  United  States  Anny  employees.  However,  the 
survey  was  sent  to  Joint  Service  Program  Offices.  All  employees  in  these  offices  must  be 
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matrixed  from  the  United  States  Army  to  support  these  efforts.  The  majority  of  the  respondents’ 
(73.4  percent)  Acquisition  Functional  Area  is  Program  Management  or  Systems  Planning, 
Research,  Development,  and  Engineering.  Respondents  included  military  officers  and  DoD 
civilians.  Three  respondents  did  not  provide  information  on  the  employment  type  and  grade. 
Most  respondents  (81  percent)  were  senior  DoD  employees,  Level  III  Acquisition  Certified  (87.5 
percent)  and  have  been  employed  by  the  DoD  for  more  than  15  years  (67.5  percent).  Participants 
have  worked  all  ACAT  level  and  non  program  of  record  efforts. 

Participants  provided  exceptional  input  to  their  knowledge  and  use  of  the  Guide 
questions.  Additionally,  many  comments  were  received  on  suggested  improvements  that  should 
be  considered.  The  research  hypotheses  are  supported.  The  Guide  process  and  procedures  do  not 
provide  an  effective  tool  for  managing  program  risk  in  today’s  acquisition  environment,  and  both 
the  literature  review  and  responses  received  from  the  research  survey  indicated  many 
improvements  that  should  be  considered  by  the  acquisition  community. 

The  DoD  acquisition  community  should  consider  updating  the  Guide  with  information 
relating  to  improved  risk  classification,  the  connection  between  risk  management  and  contract 
administration,  the  tie  between  government  and  contractor  risk  reporting  methods,  the  positive 
side  of  risk  management,  risk  management  cube  information  designed  to  specific  program  areas 
such  as  financial,  technical,  operational,  etc.,  discussion  on  program  probability  of  success, 
evaluation  of  the  program  risks,  pre -mitigated  or  post-mitigated  risk  activities,  identification  of 
root  causes  through  fishbone  diagrams  and  spider  charts,  information  on  stakeholders  and  leaders 
advocacy  and  strategic  environment  impacts,  available  training  for  risk  management 
implementation,  risks  associated  with  different  ACAT  level  programs,  and  improved  information 
on  measuring  and  tracking  risk. 

In  summary,  the  results  from  the  literature  review  and  survey  are  very  relevant  to  the 
research  due  to  the  information  identified  in  the  literature  review  and  quantity  and  quality  of 
responses  received  from  the  survey.  However,  there  are  limitations  to  the  research  and  data 
interpretation  due  to  the  survey  instrument  used.  Limitations  are  discussed  in  Chapter  5. 
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CHAPTER  5 

INTERPRETATION  AND  RECOMMENDATIONS 


Introduction 

The  intent  of  Chapter  5  is  to  provide  interpretation  of  the  results  collected  from  the 
literature  review  and  Guide  survey  and  compare  these  results  to  the  research  hypothesis. 

Chapter  5  is  structured  around  the  research  results  and  two  research  hypotheses.  A  brief 
discussion  of  the  results  is  presented.  The  research  hypotheses  are  presented  followed  by  a 
general  discussion  of  the  research  results.  Following  the  presentation  of  the  hypotheses  and 
general  discussion  is  supporting  infonnation  from  the  research.  Chapter  5  is  completed  by 
presenting  a  summary,  recommendation  for  future  research,  research  limitations, 
recommendations,  and  conclusions. 

Supporting  research  infonnation  is  presented  for  each  research  hypothesis.  Literature  and 
survey  results  are  provided  in  two  sections.  Literature  review  results  are  in  the  first  section. 
Survey  results  are  provided  in  the  second  section. 

The  survey  results  data  provided  in  the  second  section  were  designed  to  capture 
participant’s  data  on  information  provided  in  the  Guide  as  follows: 

•  General  Knowledge  and  Use 

•  Risk  Management  IPTs 

•  Program  Risk  Definition 

•  Acquisition  Program  Life  Cycle,  Risk  Update,  and  Fonnat 

•  Risk  Management  Reporting  Matrix 

•  Risk  Planning  and  Reporting 

•  Recommended  Improvements 
Research  Results 

Literature  Review  Results 

Industry  and  DoD  risk  management  topics  are  widely  available  on  most  Internet  search 
engines.  Initial  search  results  indicate  there  are  more  than  12.9  million  articles  relating  to  DoD 
risk  management.  For  industry,  the  numbers  of  Internet  hits  are  staggering.  Initial  results  indicate 
there  are  more  than  42  million  articles  for  risk  management.  However,  most  of  the  articles  for 
both  the  DoD  and  industry  are  oriented  to  the  discussion  and  application  of  risk  management. 
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Very  few  are  related  to  the  assessment  of  how  well  the  risk  management  practices  help  or  hinder 
effective  program  management.  This  study  is  limited  to  six  articles  that  discuss  the  benefits  or 
detriments  of  the  Guide. 

The  intent  of  the  literature  review  was  to  determine: 

•  Are  there  available  data  that  discusses  the  implementation  and  use  of  the  instructions 
provided  in  the  Guide? 

•  Are  there  any  benefits  for  the  use  and  implementation  of  the  instructions  provided  in  the 
Guide? 

•  Are  there  any  shortcomings  for  the  use  and  implementation  of  the  instructions  provided 
in  the  Guide? 

•  Are  there  any  suggest  improvements  to  the  Guide? 

These  answers  to  these  four  questions  are  as  follows: 

Are  there  available  data  that  discusses  the  implementation 
and  use  of  the  instructions  provided  in  the  Guide? 

Data  that  discuss  the  implementation  and  use  of  the  instructions  provided  in  the  Guide  are 
not  identified  in  the  literature  review.  Many  topics  that  discussed  the  Guide  are  included,  but 
specific  information  in  relation  to  implementation  in  a  program  environment  is  not  available. 

Are  there  any  benefits  for  the  use  and  implementation 
of  the  instructions  provided  in  the  Guide? 

The  Guide  is  a  great  basic  source  of  instruction  for  instituting  a  risk  management 
program  in  today’s  acquisition  environment.  The  Guide  is  considered  one  of  the  DoD  and 
industry  standards  for  an  effective  risk-management  program.  The  Guide  does  provide  basic 
step-by-step  instructions  for  risk  management. 

Are  there  any  shortcomings  for  the  use  and  implementation 
of  the  instructions  provided  in  Guide? 

There  is  no  information  identified  that  described  shortcomings  of  the  information 
provided  in  the  Guide.  All  information  reviewed  indicated  acceptance  of  the  data  presented  in 
the  Guide.  However,  a  number  of  suggested  improvements  were  identified. 

Are  there  any  suggest  improvements  to  the  Guide? 

•  Nominal  risk  classification  of  low,  medium,  or  high  may  be  useful  for  a  snapshot  of  risk 
status  on  a  program,  but  it  does  not  provide  enough  information  to  allocate  resources 
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(Sheppard,  2003,  p.  132).  The  current  Guide  needs  improvement  for  risk  classification. 
Information  on  allocation  of  resources  and  program  consequences  based  on  risk  would 
benefit  PMs. 

•  The  current  Guide  could  be  improved  if  it  were  to  incorporate  a  more  robust  discussion 
of  the  nexus  between  risk  management  and  contract  administration  (Bolles  2003,  p.  141). 

•  The  current  Guide  could  be  improved  if  a  discussion  of  the  relationship  between 
contractor  and  government  risk  sharing  was  included  (Bolles,  2003,  p.  144). 

•  The  current  Guide  could  be  improved  if  it  were  to  incorporate  a  discussion  on  the  tie 
between  government  and  contractor  risk-reporting  methods  (e.g.,  organizational 
responsibilities,  analyses,  products,  etc.)  (Conrow  &  Fredrickson,  1996,  pp.  6-1 1). 

•  The  current  Guide  could  be  improved  if  it  were  to  incorporate  a  discussion  on  the 
positive  side  of  risk  management.  The  Guide  should  provide  infonnation  to  prepare  for 
good  circumstances  that  can  benefit  the  program  (Frick,  2010,  p.  355). 

•  The  current  Guide  could  be  improved  if  it  were  to  incorporate  a  discussion  on 
interoperable  risk  management  (Meyers,  2006,  p.  23). 

DoD  Risk  Management  Survey  Results 

Participation  in  the  Guide  Survey  was  great.  The  survey  was  sent  to  122  individuals  and 
81  (66.3  percent)  responses  were  received.  Responses  to  each  question  in  the  survey  averaged  55 
(45  percent)  to  more  than  60  (49  percent).  This  type  of  response  to  an  Internet-based  random 
survey  is  exceptional.  This  result  may  be  attributable  to  the  subject  matter  and  interest  of  the 
DoD  acquisition  community. 

Survey  Participants  Demographics 

All  participants  identified  themselves  as  United  States  Anny  employees.  However,  the 
survey  was  sent  to  Joint  Service  Program  Offices.  All  employees  in  these  offices  must  be 
matrixed  from  the  United  States  Anny  to  support  these  efforts.  The  majority  of  the  respondents 
(73.4  percent)  Acquisition  Functional  Area  is  Program  Management  or  Systems  Planning, 
Research,  Development,  and  Engineering.  Respondents  included  military  officers  and  DoD 
civilians.  Three  respondents  did  not  provide  information  on  the  employment  type  and  grade. 
Most  respondents  (81  percent)  were  senior  DoD  employees,  Level  III  Acquisition  Certified  (87.5 
percent),  and  have  been  employed  by  the  DoD  for  more  than  15  years  (67.5  percent).  Participants 
have  worked  all  ACAT  level  and  non-program-of-record  efforts. 
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Participants  provided  exceptional  input  to  their  knowledge  and  use  of  the  Guide 
questions.  Additionally,  many  comments  were  received  on  suggested  improvements  that  should 
be  considered.  Overall,  information  provided  in  the  Guide  does  not  provide  the  acquisition 
community  an  effective  tool  in  managing  program  risk  in  today’s  acquisition  environment,  based 
on  the  responses  from  the  survey.  In  summary,  the  results  from  the  survey  are  very  relevant  to 
the  research  due  to  the  quantity  and  quality  of  responses  received. 

Research  Hypothesis  and  Discussion  of  Results 
Two  hypotheses  for  this  research  are: 

•  HI:  The  Guide  process  and  procedures  do  not  provide  an  effective  tool  in  managing 
program  risk  in  today’s  acquisition  environment. 

•  H2:  Suggested  Improvement  to  the  Guide  needs  to  be  implemented. 

HI — The  Guide  process  and  procedures  do  not  provide  an  effective  tool  in 
managing  program  risk  in  today’s  acquisition  environment. 

General  Discussion 

The  Research  Hypothesis  is  supported.  The  Guide  does  not  provide  an  effective  tool  in 
managing  program  risk  in  today’s  acquisition  environment.  Information  collected  through  the 
literature  review  supports  the  hypothesis.  Infonnation  collected  through  the  risk  management 
survey  supports  the  hypothesis. 

Supporting  Information  Literature  Review 

Literature  review  results  indicate  the  Guide  provides  a  basic  tool  for  risk  management, 
and  the  risk  management  process  is  accepted  by  government  and  industry. 

Supporting  Information  the  Guide  Survey 
General  Knowledge  and  Use 

Questions  1  and  2  from  the  Risk  Management  Survey  were  aligned  to  collect  information 
on  the  knowledge  and  use  of  the  Guide.  Question  1  was  designed  on  a  knowledge  scale  of  seven 
choices  from  not  familiar  to  very  familiar  with  the  Guide.  Each  of  the  selections  relates  to  a  14.2 
percent  increment  of  knowledge  related  to  the  Guide.  Out  of  the  81  participants  who  responded 
to  the  survey,  68  (83.9  percent)  of  the  participants  responded  to  this  question.  A  surprising  48.6 
percent  of  individuals  who  responded  to  this  question  were  less  than  somewhat  familiar  with  the 
Guide,  and  only  11.8  percent  were  very  familiar  with  the  Guide.  Question  2  was  designed  as  a 
“yes/no”  response  to  use  of  the  risk  management  process  provided  in  the  Guide.  Out  of  the  8 1 
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participants  who  responded  to  the  survey,  59  (78.2  percent)  participants  responded  to  this 
question.  Fifty-two  participants  responded  “yes”  to  this  question  and  seven  participants 
responded  “no.” 

The  results  of  these  two  questions  are  confusing.  If  less  than  half  of  the  survey  population 
was  only  somewhat  familiar  with  the  Guide,  how  can  88.1  percent  of  respondents  reply  that  they 
use  the  process  defined  in  the  Guide?  Interpretation  of  this  result  may  indicate  personnel  use 
parts  of  the  Guide  based  on  skills  they  have  learned  on  the  job  but  not  skills  they  have  learned 
through  the  Guide.  This  result  may  indicate  additional  acquisition  workforce  training  on  DoD 
risk  management  is  required. 

Risk  Management  IPTs 

Questions  3,  4,  and  5  were  aligned  to  collect  infonnation  on  Risk  Management  IPTs. 
Question  3  was  designed  to  collect  infonnation  on  risk  management  responsibility.  Out  of  the  8 1 
participants  who  responded  to  the  survey,  56  respondents  (69.1  percent)  responded  to  this 
question.  Fourteen  participants  (25  percent)  believed  risk  management  is  the  responsibility  of  the 
program  office,  and  42  respondents  (75  percent)  believed  this  is  the  responsibility  of  all 
organizations.  Question  4  is  designed  to  collect  infonnation  on  program  Risk  Management  IPTs. 
Out  of  the  81  participants  who  responded  to  the  survey,  57  respondents  (70.3  percent)  responded 
to  this  question.  Thirty-six  participants  (63.2  percent)  indicated  their  program  did  not  have  a  Risk 
Management  IPT,  and  21  (36.8  percent)  indicated  their  program  did  have  a  Risk  Management 
IPT.  Question  5  was  designed  to  collect  infonnation  on  Risk  Management  IPT  membership. 
Twenty-one  participants  responded  to  this  question,  which  follows  Question  4  response 
percentage. 

Information  provided  in  the  Guide  is  not  being  used  by  the  acquisition  community.  This 
result  may  be  related  to  participants’  lack  of  knowledge  as  reported  in  the  previous  section  of  this 
study.  IPTs  to  assess  program  risk  are  lacking,  based  on  the  participants’  responses. 
Additionally,  programs  that  have  Risk  Management  IPTs  do  not  include  critical  members  such  as 
government  contracting  personnel  as  recommended  by  the  Guide. 

Program  Risk  Definition 

Question  6,  7,  and  17  were  aligned  to  collect  information  on  program  risk  definition. 
Question  6  was  designed  to  collect  information  on  risk  management  definition.  Participants  were 
given  a  set  of  four  choices  and  asked  to  check  all  choices  that  apply.  Out  of  the  8 1  participants 
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who  responded  to  the  survey,  53  (65.4  percent)  responded  to  this  question.  All  categories  were 
selected,  and  the  other  response  category  did  not  provide  additional  relevant  information. 
Question  7  was  designed  to  collect  information  on  risk  management  tracking.  Participants  were 
given  a  set  of  three  choices  and  asked  to  check  all  choices  that  apply.  Out  of  the  8 1  participants 
who  responded  to  the  survey,  55  (67.9  percent)  responded  to  this  question.  All  categories  were 
selected,  and  participants  indicated  that  they  track  risks  for  cost,  schedule,  and  perfonnance. 
Question  17  was  designed  on  a  knowledge  scale  of  seven  choices  from  “do  not  agree”  to  “fully 
agree”  with  the  Guide.  Each  of  the  selections  relates  to  a  14.2  percent  increment  of  knowledge 
related  to  the  Guide.  Out  of  the  81  participants  who  responded  to  the  survey,  59  (72.8  percent)  of 
participants  responded  to  this  question.  Only  one  participant  fully  agreed  that  the  Guide  is 
sufficient  for  Root  Cause  Analysis  and  Risk  Identification.  Twenty-seven  (45.7  percent) 
participants  indicated  the  WBS  for  root  cause  analysis  only  somewhat  agree  with  this  approach. 

Information  provided  in  the  Guide  is  not  being  employed  by  the  acquisition  community. 
Many  responses  indicate  program  risks  are  being  associated  with  current  program  issues  and 
WBS  root  cause  analysis  is  not  an  effective  approach  for  program  risk  identification. 

Acquisition  Program  Life  Cycle,  Risk  Update,  and  Format 

Question  8,  9,  and  10  were  aligned  to  collect  infonnation  on  acquisition  program  life 
cycle,  frequency  of  program  risk  update,  and  risk  reporting  fonnats  used  by  participants.  These 
questions  did  not  directly  relate  to  information  provided  in  the  Guide.  These  questions  were 
asked  to  gain  infonnation  on  the  survey  participants  and  their  approach  to  risk  updates  and 
formats  used  for  reporting. 

Risk  Management  Reporting  Matrix 

Question  11,  12,  13,  14,  and  15  were  aligned  to  collect  infonnation  on  the  risk 
management  reporting  matrix.  Question  1 1  was  designed  on  a  knowledge  scale  of  7  choices  from 
“does  not  provide”  to  fully  portray  an  accurate  and  fair  portrayal  of  program  risk  based  on 
likelihood  and  consequence.”  Each  of  the  selections  relates  to  a  14.2  percent  increment  of 
knowledge  related  to  the  Guide.  Out  of  the  81  participants  who  responded  to  the  survey,  56  (69.1 
percent)  participants  responded  to  this  question.  No  participants  believed  the  Risk  Reporting 
Matrix  fully  portrays  program  risk.  Fifteen  (26.7  percent)  participants  indicated  the  Risk 
Management  Matrix  provides  minimal  portrayal  of  program  risk.  Question  12  was  designed  on  a 
knowledge  scale  of  seven  choices  ranging  from  “does  not  provide”  to  fully  provide  likelihood  of 
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risk  likelihood  and  probability  of  occurrence.”  Each  of  the  selections  relates  to  a  14.2  percent 
increment  of  knowledge  related  to  the  Guide.  Out  of  the  8 1  participants  who  responded  to  the 
survey,  55  (67.9  percent)  participants  responded  to  this  question.  No  participants  believed  the 
risk  likelihood  provides  a  fair  assessment.  Fourteen  (25.4  percent)  of  the  participants  indicated 
the  risk  likelihood  and  probability  of  occurrence  provides  a  minimal  portrayal  of  program  risk. 
Question  13  was  designed  as  a  “yes/no”  response  to  modification  or  customization  on  the  level 
of  likelihood  for  a  specific  program.  Out  of  the  8 1  participants  who  responded  to  the  survey,  63 
(77.7  percent)  of  participants  responded  to  this  question.  Fifty-three  participants  responded  “no” 
to  this  question,  and  10  responded  “yes.”  Question  14  was  designed  on  a  knowledge  scale  of 
seven  choices  from  “does  not  provide”  to  “fully  portray  an  accurate  and  fair  portrayal  of  levels 
and  type  of  consequence  criteria.”  Each  of  the  selections  relates  to  a  14.2  percent  increment  of 
knowledge  related  to  the  Guide.  Out  of  the  81  participants  who  responded  to  the  survey,  61  (75.3 
percent)  of  participants  responded  to  this  question.  No  participants  believed  consequence  criteria 
provide  an  accurate  or  fair  portrayal.  Question  15  was  designed  as  a  “yes/no”  response  to 
modification  or  customization  on  the  consequence  criteria  for  a  specific  program.  Out  of  the  8 1 
participants  who  responded  to  the  survey,  62  (76.5  percent)  participants  responded  to  this 
question.  Fifty-two  participants  responded  “no”  to  this  question  and  10  responded  “yes.” 

The  risk  management  reporting  matrix  provided  in  the  Guide  does  not  provide  the 
acquisition  community  a  good  tool  for  risk  management  according  to  participant’s  survey 
responses.  In  most  cases,  25  percent  or  more  of  the  participants  believe  the  matrix  provides 
minimal  risk  infonnation. 

Risk  Planning  and  Reporting 

Question  18,  19,  and  20  were  aligned  to  collect  information  on  risk  planning  and 
reporting.  Question  18  was  designed  on  a  knowledge  scale  of  seven  choices  from  “does  not 
follow  the  format”  to  “fully  follows  the  format  of  the  risk  mitigation  plan  infonnation  provided 
in  the  guide.”  Each  of  the  selections  relates  to  a  14.2  percent  increment  of  knowledge  related  to 
the  Guide.  Out  of  the  81  participants  who  responded  to  the  survey,  55  (67.9  percent)  participants 
responded  to  this  question.  Four  participants  fully  follow  the  fonnat  provided  in  the  Guide. 
Twenty-six  (45.2  percent)  of  participants  indicated  their  risk  mitigation  plan  only  uses  part  of  the 
Guide  format.  Question  19  was  designed  on  a  knowledge  scale  of  seven  choices  from  “does  not 
follow  the  format”  to  “fully  follows  the  fonnat  of  the  risk  management  plan  information 
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provided  in  the  guide.”  Each  of  the  selections  relates  to  a  14.2  percent  increment  of  knowledge 
related  to  the  Guide.  Of  the  81  participants  who  responded  to  the  survey,  58  (71.6  percent) 
participants  responded  to  this  question.  Five  participants  fully  follow  the  fonnat  provided  in  the 
Guide.  Twenty-eight  (48.2  percent)  participants  indicated  their  risk  management  plan  uses  only 
part  of  the  Guide  format.  Question  20  was  designed  on  a  knowledge  scale  of  seven  choices  from 
“does  not  provide”  to  “provides  all  infonnation  to  management  to  make  timely  and  accurate 
decisions.”  Each  of  the  selections  relates  to  a  14.2  percent  increment  of  knowledge  related  to  the 
Guide.  Out  of  the  81  participants  who  responded  to  the  survey,  61  (75.3  percent)  participants 
responded  to  this  question.  Only  one  participant  believes  the  reporting  process  provides 
management  infonnation  to  make  timely  and  accurate  decisions.  Thirty  (49.1  percent) 
participants  indicated  the  reporting  process  only  provides  some  infonnation  to  management. 

Recommended  Improvements 

Questions  2 1  and  22  were  aligned  to  collect  information  on  other  risk  management  tools 
in  use  by  participants  and  recommended  improvements.  Other  tools  were  not  identified  and 
recommended  improvements  are  discussed  later  in  this  chapter. 

H2 — Suggested  Improvement  to  the  Guide  needs  to  be  implemented. 

General  Discussion 

The  research  hypotheses  are  supported.  Improvements  to  the  Guide  need  to  be 
implemented  and  both  the  literature  review  and  responses  received  from  the  research  survey 
indicated  many  improvements  that  should  be  considered  by  the  acquisition  community. 

Supporting  Information  Literature  Review 
The  literature  review  provided  six  suggested  improvements  as  follows: 

1 .  Nominal  risk  classification  of  low,  medium,  or  high  may  be  useful  for  a  snapshot  of  risk 
status  on  a  program,  but  it  does  not  provide  enough  infonnation  to  allocate  resources. 
(Sheppard,  2003,  p.  132).  The  current  Guide  needs  improvement  for  risk  classification. 
Infonnation  on  allocation  of  resources  and  program  consequences  based  on  risk  would 
benefit  PMs. 

2.  The  cunent  Guide  could  be  improved  if  it  were  to  incorporate  a  more  robust  discussion 
of  the  nexus  between  risk  management  and  contract  administration  (Bolles  2003,  p.  141). 

3.  The  cunent  Guide  could  be  improved  if  a  discussion  of  the  relationship  between 
contractor  and  government  risk  sharing  was  included  (Bolles,  2003,  p.  144). 
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4.  The  current  Guide  could  be  improved  if  it  were  to  incorporate  a  discussion  on  the  tie 
between  government  and  contractor  risk  reporting  methods  (e.g.,  organizational 
responsibilities,  analyses,  products,  etc.)  (Conrow  &  Fredrickson,  1996,  pp.  6-1 1). 

5.  The  current  Guide  could  be  improved  if  it  were  to  incorporate  a  discussion  on  the 
positive  side  of  risk  management.  The  Guide  should  provide  infonnation  to  prepare  for 
good  circumstances  that  can  benefit  the  program  (Frick,  2010,  p.  355). 

6.  The  current  Guide  could  be  improved  if  it  were  to  incorporate  a  discussion  on 
interoperable  risk  management  (Meyers,  2006,  p.  23). 

Supporting  Information  the  Guide  Survey 

The  Guide  Survey  resulted  in  many  suggested  improvements  to  the  Guide.  Many 
repetitive  comments  were  received  from  the  participants.  Ten  possible  improvements  were 
identified  as  follows: 

1.  The  current  Guide  could  be  improved  if  the  risk  management  cube  infonnation  were 
designed  to  specific  program  areas  such  as  financial,  technical,  operational,  etc. 

2.  The  current  Guide  could  be  improved  if  a  discussion  on  program  probability  of  success 
were  included. 

3.  The  current  Guide  could  be  improved  if  infonnation  is  provided  on  evaluation  of  the 
program  risks. 

4.  The  current  Guide  could  be  improved  if  information  is  provided  on  pre-mitigated  or  post- 
mitigated  risk  activities. 

5.  The  cunent  Guide  could  be  improved  if  information  is  provided  on  risk  associated  with 
programs  in  different  stages  of  the  life  cycle. 

6.  The  cunent  Guide  could  be  improved  if  infonnation  is  provided  on  identifying  root 
causes  through  fishbone  diagrams  and  spider  charts. 

7.  The  cunent  Guide  could  be  improved  if  infonnation  is  provided  on  stakeholders  and 
leaders  advocacy  and  strategic  environment  impacts. 

8.  The  cunent  Guide  could  be  improved  if  information  is  provided  on  available  training  for 
risk  management  implementation. 

9.  The  current  Guide  could  be  improved  if  information  is  provided  on  risks  associated  with 
different  AC  AT  level  programs. 
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10.  The  current  Guide  could  be  improved  if  information  is  provided  on  measuring  and 
tracking  risk. 

Summary 

The  Strategy  Research  Project  consists  of  the  review  of  the  Guide  and  risk  management 
documentation  and  articles  from  multiple  Internet  sources.  Review  and  presentation  of  the  Guide 
was  conducted  to  provide  the  reader  of  this  report  a  general  understanding  of  DoD  risk 
management  practices.  Risk  management  documentation  and  articles  provide  an  understanding 
of  the  effectiveness  and  usefulness  of  risk  management  practices. 

Additionally,  this  study  includes  a  survey  to  understand  risk  management  practices 
currently  in  use  by  the  DoD  acquisition  community.  The  survey  was  aligned  to  gather  data  on 
knowledge  and  relevance  of  the  Guide,  respondent  demographics,  and  other  risk  management 
tools  currently  in  use  by  the  acquisition  community. 

This  study  was  aligned  to  applied  research  and  the  solution  of  issues.  This  study  was 
conducted  for  the  purpose  of  evaluating  ongoing  instructions  for  DoD  acquisition  program  risk 
management.  This  study  follows  the  descriptive  research  method.  A  review  of  available 
applicable  data  and  a  survey  to  address  the  research  question  and  research  hypothesis  were 
included.  The  study  is  designed  to  measure  the  effectiveness  of  current  policy  in  relation  to  risk 
management. 

The  study  identified  the  Guide  provides  a  basic  tool  for  risk  management  and  the  Guide 
is  accepted  by  government  and  industry.  The  Guide  does  not  provide  an  effective  tool  for 
management  of  the  wide  variety  of  projects  ongoing  in  the  DoD  acquisition  environment.  Many 
recommended  improvements  to  the  Guide  were  identified  through  this  research.  Other  risk 
management  tools  currently  in  use  by  the  acquisition  community  were  not  identified  in  this 
research. 

Recommendations  for  Future  Research 

This  study  provides  the  start  of  Guide  analysis.  Future  researchers  can  build  on  this  study 
in  many  ways  to  include: 

•  Analysis  of  risk  management  requirements  and  needs  of  personnel  in  each  of  the 
acquisition  functional  areas. 

•  Analysis  of  risk  management  requirements  and  needs  of  personnel  in  different  acquisition 
category  program  efforts. 
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•  Application  of  tools  such  as  Monte  Carlo,  fishbone  diagrams,  and  spider  charts  to  DoD 
risk  management  practices. 

•  Required  and  applicable  risk  management  training  programs  for  the  DoD  acquisition 
community. 

•  Research  to  address  strategic  risk  management  in  relation  to  political  environments  and 
projected  DoD  funding  levels. 

•  Allocation  of  program  resources  for  risk  management. 

•  Successful  ties  between  government  and  industry  risk  management. 

Research  Limitations 

Literature  Review 

Industry  and  DoD  risk  management  topics  are  widely  available  on  most  Internet  search 
engines.  Initial  search  results  indicate  there  are  more  than  12.9  million  articles  relating  to  DoD 
risk  management.  For  industry,  the  numbers  of  Internet  hits  are  staggering.  Initial  results  indicate 
there  are  more  than  42  million  articles  for  risk  management.  However,  most  of  the  articles  for 
both  the  DoD  and  Industry  are  oriented  to  the  discussion  and  application  of  risk  management. 
Very  few  are  related  to  the  assessment  of  how  well  the  risk  management  practices  help  or  hinder 
effective  program  management.  This  study  is  limited  to  a  few  articles  that  discuss  the  benefits  or 
detriments  of  the  Guide. 

Research  Questionnaire 

Considering  there  are  hundreds  of  thousands  of  members  of  the  acquisition  workforce 
programs,  there  is  a  limit  to  the  number  of  individual  surveys  that  can  be  analyzed.  The  survey 
was  designed  to  accommodate  input  from  many  acquisition  community  members’  across 
functional  fields  but  is  only  a  select  sample  of  all  involved  with  the  acquisition  of  products  for 
the  Joint  Services. 

Survey  Instrument 

The  Internet  survey  tool  used  for  this  research  provided  an  effective  way  to  gain 
comments  from  the  acquisition  community.  However,  the  tool  only  will  allow  the  user  to  gain 
compiled  data  in  relation  to  the  question.  The  tool  does  not  permit  detailed  analysis  of  the  results 
based  on  single  respondent  infonnation  or  a  group  of  respondents.  For  instance,  sorting  and 
analyzing  infonnation  on  respondents  related  to  acquisition  career  field  or  ACAT  program  were 


53 


not  permitted  through  use  of  this  tool.  This  type  of  infonnation  may  have  permitted  an 
assessment  of  the  Guide  to  functional  area  or  ACAT  level  program  efforts. 

Recommendations 

The  DoD  acquisition  community  should  consider  updating  the  Guide  with  information 
relating  to  improved  risk  classification,  nexus  between  risk  management  and  contract 
administration,  tie  between  government  and  contractor  risk  reporting  methods,  positive  side  of 
risk  management,  risk  management  cube  infonnation  designed  to  specific  program  areas  such  as 
financial,  technical,  operational,  etc.,  discussion  on  program  probability  of  success,  evaluation  of 
the  program  risks,  pre-mitigated  or  post-mitigated  risk  activities,  identification  of  root  causes 
through  fishbone  diagrams  and  spider  charts,  information  on  stakeholders  ‘and  leaders’  advocacy 
and  strategic  environment  impacts,  available  training  for  risk  management  implementation,  risks 
associated  with  different  ACAT  level  programs,  and  improved  infonnation  on  measuring  and 
tracking  risk. 

Conclusions 

The  Guide  provides  acquisition  personnel  a  basic  guide  for  the  assessment  and 
presentation  of  program  risks.  The  Guide  provides  a  process  for  risk  management,  key  activities 
for  risk  identification,  analysis  and  mitigation,  and  information  on  risk  planning  and  preparation 
activities  for  risk  management. 

The  Guide  is  the  DoD’s  effort  to  establish  a  baseline  for  the  management  and  reporting  of 
program  risk.  Overall,  the  Guide  presents  a  risk  management  matrix  that  projects  program  risk 
based  on  levels  of  likelihood  and  consequence  criteria. 

The  Guide  and  management  matrix  are  used  by  many  DoD  acquisition  programs  and 
presented  at  many  meeting  to  senior  level  leaders  as  the  tool  for  risk  management.  Over  the  past 
years,  during  multiple  meetings  and  program  reviews,  risk  reporting  in  accordance  with  the 
Guide  has  come  into  question  by  many  individuals  at  meetings  or  program  reviews.  Questions 
concerning  the  applicability,  reliability,  and  accuracy  of  the  data  presented  based  on  the  Guide 
can  sometimes  relate  to  the  process  for  risk  management  opposed  to  program  related  risks. 

Risk  management  is  critical  for  program  success.  Presentation  and  understanding  of 
program  risks  by  all  individuals  who  have  concern  about  the  program  is  also  critical  for  program 
success.  Having  the  right  tool,  at  the  right  time  that  allows  full  definition  and  understanding  of 
program  risk  is  critical  for  successful  program  management. 
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The  Strategy  Research  Project  consisted  of  the  review  of  the  Guide  and  risk  management 
documentation  and  articles  from  multiple  Internet  sources.  Review  and  presentation  of  the  Guide 
is  conducted  to  provide  the  reader  of  this  report  a  general  understanding  of  DoD  risk 
management  practices.  Risk  management  documentation  and  articles  provide  an  understanding 
of  the  effectiveness  and  usefulness  of  risk  management  practices. 

Additionally,  this  study  included  a  survey  to  understand  risk  management  practices 
currently  in  use  by  the  DoD  acquisition  community.  The  survey  was  aligned  to  gather  data  on 
knowledge  and  relevance  of  the  Guide,  respondent  demographics,  and  other  risk  management 
tools  currently  in  use  by  the  acquisition  community. 

The  study  identified  the  Guide  provides  a  basic  tool  for  risk  management  and  the  Guide 
is  accepted  by  government  and  industry.  The  Guide  does  not  provide  an  effective  tool  for 
management  of  the  wide  variety  of  projects  ongoing  in  the  DoD  acquisition  environment.  Many 
recommended  improvements  to  the  Guide  were  identified  through  this  research.  Other  risk 
management  tools  currently  in  use  by  the  acquisition  community  were  not  identified  in  this 
research. 
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GLOSSARY  OF  ACRONYMS  AND  TERMS 


ACAT 

Acquisition  Category 

AIS 

Automated  Information  System 

DAU 

Defense  Acquisition  University 

DoD 

Department  of  Defense 

ER 

Environmental  Restoration 

EVM 

Earned  Value  Management 

FAR 

Federal  Acquisition  Regulation 

FY 

Fiscal  Year 

IMS 

Integrated  Master  Schedule 

IPT 

Integrated  Product  Team 

KPP 

Key  Performance  Parameter 

LCC 

Life  Cycle  Cost 

MILCON 

Military  Construction 

WBS 

Work  Breakdown  Structure 
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APPENDIX  A 

RISK  MANAGEMENT  GUIDE  FOR  DOD  ACQUISITION  OVERVIEW 

Risk  Management  Guide  for  DoD  Acquisition  Overview 

The  Guide  provides  acquisition  personnel  a  basic  guide  for  the  assessment  and 
presentation  of  program  risks.  The  Guide  provides  a  process  for  risk  management,  key  activities 
for  risk  identification,  analysis  and  mitigation,  and  information  on  risk  planning  and  preparation 
activities  for  risk  management. 

The  Guide  is  the  DoD  effort  to  establish  a  baseline  for  the  management  and  reporting  of 
program  risk.  Overall,  the  Guide  presents  a  risk  management  matrix  that  programs  risk  based  on 
levels  of  likelihood  and  consequence  criteria. 

The  Guide  defines  program  risk  as: 

Risk  is  a  measure  of  future  uncertainties  in  achieving  program  performance  goals  and 
objectives  within  defined  cost,  schedule,  and  performance  constraints.  Risk  can  be  associated 
with  all  aspects  of  a  program  (e.g.,  threat,  technology  maturity,  supplier  capability,  design 
maturation,  performance  against  plan,)  as  these  aspects  relate  across  the  Work  Breakdown 
Structure  (WBS)  and  Integrated  Master  Schedule  (IMS).  Risk  addresses  the  potential  variation  in 
the  planned  approach  and  its  expected  outcome.  While  such  variation  could  include  positive  as 
well  as  negative  effects,  this  guide  will  only  address  negative  future  effects  since  programs  have 
typically  experienced  difficulty  in  this  area  during  the  acquisition  process  (DoD,  2006). 

Risks  have  three  components: 

•  A  future  root  cause  (yet  to  happen ),  which,  if  eliminated  or  corrected,  would  prevent  a 
potential  consequence  from  occurring, 

•  A  probability  (or  likelihood )  assessed  at  the  present  time  of  that  future  root  cause 
occurring,  and 

•  The  consequence  (or  effect)  of  that  future  occurrence. 

A  future  root  cause  is  the  most  basic  reason  for  the  presence  of  a  risk.  Accordingly,  risks  should 
be  tied  to  future  root  causes  and  their  effects  (DoD,  2006). 

The  Guide  further  defines  Program  Risk  Management  and  includes  a  risk  management 
model  that  states  program  risk  has  five  key  activities  and  is  performed  throughout  the  program 
life  cycle: 
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Risk  management  is  a  continuous  process  that  is  accomplished  throughout  the  life  cycle 
of  a  system.  It  is  an  organized  methodology  for  continuously  identifying  and  measuring  the 
unknowns;  developing  mitigation  options;  selecting,  planning,  and  implementing  appropriate 
risk  mitigations;  and  tracking  the  implementation  to  ensure  successful  risk  reduction.  Effective 
risk  management  depends  on  risk  management  planning;  early  identification  and  analyses  of 
risks;  early  implementation  of  corrective  actions;  continuous  monitoring  and  reassessment;  and 
communication,  documentation,  and  coordination  (DoD,  2006). 

Acquisition  program  risk  management  is  not  a  stand-alone  program  office  task.  It  is 
supported  by  a  number  of  other  program  office  tasks.  In  turn,  the  results  of  risk  management  are 
used  to  finalize  those  tasks.  Important  tasks,  which  must  be  integrated  as  part  of  the  risk 
management  process,  include  requirements  development,  logical  solution  and  design  solution 
(systems  engineering),  schedule  development,  performance  measurement,  EVM  [Earned  Value 
Management]  (when  implemented),  and  cost  estimating.  Planning  a  good  risk  management 
program  integral  to  the  overall  program  management  process  ensures  risks  are  handled  at  the 
appropriate  management  level  (DoD,  2006). 

Emphasis  on  risk  management  coincides  with  overall  DoD  efforts  to  reduce  life-cycle 
costs  (LCC)  of  system  acquisitions.  New  processes,  reforms,  and  initiatives  are  being 
implemented  with  risk  management  as  a  key  component.  It  is  essential  that  programs  define, 
implement,  and  document  an  appropriate  risk  management  and  mitigation  approach.  Risk 
management  should  be  designed  to  enhance  program  management  effectiveness  and  provide 
PMs  with  a  key  tool  to  reduce  LCC,  increase  program  likelihood  of  success,  and  assess  areas  of 
cost  uncertainty  (DoD,  2006). 

The  risk  management  process  model  (see  Figure  1)  includes  the  following  key  activities, 
performed  on  a  continuous  basis: 

•  Risk  Identification, 

•  Risk  Analysis, 

•  Risk  Mitigation  Planning, 

•  Risk  Mitigation  Plan  Implementation,  and 

•  Risk  Tracking  (DoD,  2006). 
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Figure  1.  DoD  Risk  Management  Process 


The  Guide  provides  a  Risk  Management  Cube  (Figure  2)  for  the  presentation  of  program 
risk  that  portrays  risk  in  red,  yellow,  and  green.  These  ratings  allow  a  quick  visual  assessment  of 
program  risk.  A  red  rating  indicates  high  risk  to  the  program,  yellow  ratings  indicate  moderate 
risk  to  the  program,  and  green  risk  identifies  low  program  risks.  The  risk  management  cube  is 
populated  through  the  use  of  a  level  of  likelihood  criteria  (Figure  3)  and  consequence  criteria 
(Figure  4). 

Each  undesirable  event  that  might  affect  the  success  of  the  program  (performance, 
schedule,  and  cost)  should  be  identified  and  assessed  as  to  the  likelihood  and  consequence  of 
occurrerice.  A  standard  format  for  evaluation  and  reporting  of program  risk  assessment  findings 
facilitates  common  understanding  of  program  risks  at  all  levels  of  management.  The  Risk 
Reporting  Matrix  below  is  typically  used  to  determine  the  level  of  risks  identified  within  a 
program.  The  level  of  risk  for  each  root  cause  is  reported  as  low  (green),  moderate  (yellow),  or 
high  (red)  (DoD,  2006). 
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Consequence 


Figure  2.  Risk  Reporting  Matrix 

The  level  of  likelihood  of  each  root  cause  is  established  utilizing  specified  criteria 
(Figure  3).  For  example,  if  the  root  cause  has  an  estimated  50  percent  probability  of  occurring, 
the  corresponding  likelihood  is  Level  3  (DoD,  2006). 


Level 

Likelihood 

Probability  of  Occurrence 

1 

Not  Likely 

-10% 

2 

Low  Likelihood 

-30% 

3 

Likely 

-50% 

4 

Highly  Likely 

-70% 

5 

Near  Certainty 

-90% 

Figure  3.  Levels  of  Likelihood  Criteria 
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Level 

Technical  Performance 

Schedule 

Cost 

1 

Minimal  or  no  consequence  to  technical 
performance 

Minimal  or  no  impact 

Minimal  or  no 
impact 

2 

Minor  reduction  in  technical  performance  or 
supportability,  can  be  tolerated  with  little  or  no 
impact  on  program 

Able  to  meet  key  dates. 

Slip  <  _f_  month(s) 

Budget  increase  or 
unit  production  cost 
increases. 

<  **  (1%  of 
Budget) 

3 

Moderate  reduction  in  technical  performance  or 
supportability  with  limited  impact  on  program 
objectives 

Minor  schedule  slip.  Able 
to  meet  key  milestones 
with  no  schedule  float. 

Slip  <  month(s) 
Sub-system  slip  > 
month(s)  plus  available 
float. 

Budget  increase  or 
unit  production  cost 
increase 

<  **  (5  %  of 

Budget) 

4 

Significant  degradation  in  technical  performance  or 
major  shortfall  in  supportability;  may  jeopardize 
program  success 

Program  critical  path 
affected. 

Slip  <  _f_  months 

Budget  increase  or 
unit  production  cost 
increase 
<  **  (10%  of 
Budget) 

5 

Severe  degradation  in  technical  performance; 
Cannot  meet  KPP  or  key  technical/supportability 
threshold;  will  jeopardize  program  success 

Cannot  meet  key  program 
milestones. 

Slip  >  _f_  months 

Exceeds  APB 
threshold 
>  **  (10%  of 
Budget) 

Figure  4.  Levels  and  Type  of  Consequence  Criteria 

The  level  and  types  of  consequences  of  each  risk  are  established  utilizing  criteria  such  as 
those  described  in  Figure  4.  A  single  consequence  scale  is  not  appropriate  for  all  programs, 
however.  Continuing  with  the  prior  example  of  a  root  cause  with  a  50  percent  probability  of 
occurring,  if  that  same  root  cause  has  no  impact  on  performance  or  cost,  but  may  likely  result  in 
a  minor  schedule  slippage  that  won  7  impact  a  key  milestone,  then  the  corresponding 
consequence  is  a  Level  3  for  this  risk.  For  clarity,  it  is  also  classified  as  a  schedule  risk  since  its 
root  cause  is  schedule  related  (DoD,  2006). 
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